Kopernik - Ezgi Cakir

Week One

Title: Meeting with Prof. Jones

Date: 09/07/2020

Subject: Overview and Planning of Kopernik

Attendance: Ezgi Cakir, Prof. Merwyn Jones

Meeting Agenda

- Introduced to necessary tools such as Selenium and Raspberry Pi. Selenium will be used to control Wordpress which deals with Kopernik website. After getting photos of the satellite during 24 hours, new photos must be posted on the Kopernik's website. Selenium through Python will organize how the photos will be posted and updated/deleted.

- It will be possible to access Raspberry Pi with VNC Viewer remotely whereas, when needed Raspberry Pi could be used manually to change the source code. When necessary Prof. Jones will lend me a Raspberry Pi. In order to connect to Raspberry Pi interface, Andrew Deskur will provide me the credentials (userid and password).

- One of the current problems within the project is the chrome file does not work properly after the microSD card got burned. Chrome file needs to be rebooted for the root and the user. Since amplifier between the satellite and Raspberry Pi depends on it, it has a high priority.

- For future improvements, the photos of last 7 days of satellites could be saved on the website. Currently, it displays the recent photos while deleting the old ones from the website.

Current Status

- Meeting with Andrew Deskur will be scheduled to discuss the details and requirements of Kopernik. Till that meeting, available material and research on the project will be done.

- Prof. Jones has shared possible resources to help me get started on the project research and understanding wise. I will be checking those resources to prepare the meeting with Andrew Deskur.

- CPT form is yet to be processed to ISSS to get started with the project in terms of source code and access.

- Time for weekly meetings with Prof. Jones is scheduled. Every week at that time I will be providing status update and insight about the project. Time of the weekly meetings is titled to change dependent on Mr. Deskur's schedule.

Accomplishments

- I have filled out the CPT form and sent it to Prof. Jones.

- I have contacted to Mr. Deskur to set up a meeting to discuss further details on the project.

- I got my access to edit and create on Wiki.

Week Two

Title: Weekly Development

Date: 09/14/2020

Subject: Access to Kopernik System

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Weekly Progress

· Meeting is canceled or postponed due to CPT form delay. The status of the meeting of this week depends on the process. This update includes the written communication of attendants.

- Set up VNC Viewer to access Raspberry Pi remotely. There was a drawback about my access being confirmed. I have contacted Prof Jones and Andrew Deskur. Andrew has given me additional credentials to confirm my access. I have logged in to the email address to confirm. Got familiar with Raspberry Pi interface as well as VNC Viewer. - The BiasT has to be booted every time the system starts. Right now, booting takes place manually because there is a problem with the chrontab. Learned the right way to access the file and started to figure out a way to debug it. For scripts, there are several IDEs or tools to debug. Currently working on the best way or the possible way to debug the chron file to boot automatically.

- Filled out the CPT electronic form in the ISSS page. Requested an express mail and made the payment. Added Prof Jones and Mr. Deskur to email notifications to let them know about the new I-20 process. Agreed on rescheduling the weekly meeting after I-20 arrives.

Current Status

- Learning and getting accustomed with Raspberry Pi and VNC Viewer interfaces. I examine the source codes without making any changes to figure out how it is working and what the problems are. Chron files and script debugging in this sense are prioritized. Going through the resources Prof Jones provided me last week to have a clear sense of the project requirements.

- Waiting for new I-20 to arrive to get started with the project officially.

Accomplishments

- I have completed Electronic request for CPT form to ISSS as well as requesting an express mail. I have provided necessary information such as address, delivery option and payment details to create a shipping ID ticket. I have emailed the shipping ID to ISSS and confirmed that they got the email.

- Mr. Deskur will be attending the weekly meetings with Prof Jones every Monday mornings to discuss developments in the course of the project. This week’s meeting is likely to be rescheduled due to CPT form.

- I could access Raspberry Pi through VNC Viewer remotely. Currently I only use it to examine the code and nothing else.

Week Three

Title: Weekly Meeting

Date: 09/21/2020

Subject: Issue with Cron File

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- We have discussed the possible reasons why the Cron file stopped working after SD card got burned. I got filled in the tried solutions so far to have an appropriate starting point to solve the problem. Cron file has been deleted and recreated the same way in case the file got damaged. However, the same problem kept going on.

- Mr. Deskur recommended me an app to keep track of the times that satellites send the images. Thanks to that, I will be able to comprehend if the system is working right or not. During those hours when the system is booted, I am supposed to be getting clear images with no noise.

- Some of the sensors of the satellites do not work properly with the current parameter. Most probably the parameter of the bandwidth will be changed in RPI to also display red and green colors. In order to achieve that, I will be working on how to set the frequency right.

Current Status

- I got clarification on Cron files. The current Cron file consists of two lines in which line 1 does not work while line 2 works. It also came to my attention that there are two Cron files: one for the root and the other for the user. It is not specific which one BiasT uses to reboot itself. Our guess is it is using both of them combined. Next week I’ll be working on debugging the Cron file.

- Mr. Deskur added a new improvement to the improvements. On the Kopernik website where the satellite images are displayed, sometimes there are 2-3 of the same thumbnails and only one of those thumbnails opens when clicked on. Mr. Deskur wishes the repeated thumbnails fixed.

Accomplishments

- I have my new I-20 which means I will be able to update/change the source code of the project from on. The current version of the project is backed up for any disastrous possibility.

- I found several resources on how to debug the crontab file to fix the issue of rebooting the BiasT automatically. I got familiar with using the VNC Viewer and the whereabouts of the source files. Started to use an app to know the exact time when satellites NOAA-15, NOAA-18 and NOAA-19 will send photos.

- I went over the resources Prof. Jones and Andrew Deskur provided me. For the moment, I will be using VNC Viewer to access to RPI remotely. In case of need, I might build my own system to work on the project.

Week Four

Title: Weekly Meeting

Date: 09/28/2020

Subject: Issue with Cron File

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- I have been researching to find a way to debug the cron file. In order to see how it executes, I tried to execute it from the terminal. I got the following error: Failed to execute child process “xterm” (No such file or directory). Thus, I ran the command “sudo apt-get install xterm to complete the missing process.

- When the command line “sed ‘s.. /var/log/boot” is ran, we get the “Nothing has been logged yet. If you’re still seeing this message your current init system might not write bootup messages to the system console at all” error. This error supports the log problem I have encountered on the Internet during my research. I will be looking at this for the next week.

Current Status

- The relationship of cron file and rebooting had made clear for me. Mostly cron files have a schedule that they run. In this project, the cron file is expected to run every time the system restarts. As far as I know, there are other types of files that make it possible and more efficient. If cron file problem cannot be averted, I might choose a different approach to solve it. - The error I fixed so far does not solve the cron file crisis on its own. The images we get from the satellites are still distorted. BiasT has to reboot itself automatically for the images to be clear of noise.

Accomplishments

- The error about xterm file I was getting is fixed. The terminal window at the bottom right opens and closes. Prof. Jones directed me to how to keep the window open.

- Since the cron file only consists of two lines of script, I suspect that the issue has got to do with booting files not being logged or a PATH issue. Drew helped me figure out possible ways to monitor the cron file activities to specify the problem. There could be several command lines I could use to fix the problem.

- I found tools like Cronitor that monitors the activities of the cron file and notifies you. There is a trial version available. It could be used as a last resource to debug.

Week Five

Title: Weekly Meeting

Date: 10/05/2020

Subject: Issue with Cron File

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- I have been trying to send the log files of crontab to syslog. When logs are kept and sent, there is a good chance to figure out what is the issue. Crontab permissions and changing the frequency of the crontab are my priorities to debug the crontab. I have located several logs which I could not make sense yet.

- There was an issue with the website that prevented it from working. Apart from BiasT not getting the images, the website was down. The cause is unknown.

Current Status

- There are multiple cron file logs which include one ‘pi’ and many ‘root’ logs. I am trying to make sense of what that means. Originally, there was supposed to be one user and one root crontab. - The images from the satellite remain still distorted which means that BiasT is still not working properly.

Accomplishments

- After taking back some of the changes the script of the website is running. The table of logs is as follows:

pi@raspberrypi:~/rtl_biast/build/src $ grep CRON /var/log/syslog Oct 4 00:02:01 raspberrypi CRON[18029]: (pi) CMD (/home/pi/weather/predict/schedule_all.sh) Oct 4 00:17:01 raspberrypi CRON[19117]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 01:17:02 raspberrypi CRON[20949]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 02:17:01 raspberrypi CRON[23033]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 03:17:01 raspberrypi CRON[24613]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 04:17:01 raspberrypi CRON[26390]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 05:17:01 raspberrypi CRON[28326]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 06:17:01 raspberrypi CRON[30258]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 06:25:01 raspberrypi CRON[30486]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )) Oct 4 06:47:02 raspberrypi CRON[31214]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )) Oct 4 07:17:01 raspberrypi CRON[32137]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 08:17:01 raspberrypi CRON[2564]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 09:17:01 raspberrypi CRON[4451]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 10:17:01 raspberrypi CRON[6926]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 11:17:01 raspberrypi CRON[8934]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 12:17:01 raspberrypi CRON[11181]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 13:17:02 raspberrypi CRON[13051]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 14:17:01 raspberrypi CRON[14868]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 15:17:01 raspberrypi CRON[16702]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 16:17:01 raspberrypi CRON[18558]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 17:17:01 raspberrypi CRON[20527]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 18:17:01 raspberrypi CRON[23086]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 19:17:01 raspberrypi CRON[24852]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 20:17:02 raspberrypi CRON[27606]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 21:17:01 raspberrypi CRON[30389]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 22:17:01 raspberrypi CRON[32185]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 4 23:17:01 raspberrypi CRON[2579]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

The root crontab keeps trying to execute itself but pi crontab tries to execute only once. If they are both supposed to be working at the same time, I might have found which crontab has the issue.

Week Six

Title: Weekly Meeting

Date: 10/12/2020

Subject: BiasT

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- In order to get clear images, BiasT needs to be rebooted automatically. Cron file is responsible to make that schedule happen. For the past weeks, I have been trying to figure out the issue with the cron file and how to debug it. My two suspicions were a PATH issue or log problem. A recent debugging made it clear that it was a log problem.

Current Status

- Ever since I have run a command about the cron file tracking down the logs, the images are clear. BiasT processes the images that are collected from satellites. Two cron files reboot themselves at the same time at midnight.

- Prof Jones and Andrew will direct me what to do next at tomorrow’s meeting.

Accomplishments

- BiasT reboots itself automatically which was the first and the most important problem to solve. - Sensors collect green and red colors as well which was one of the problems Drew mentioned when I took the requirements of the project.

Week Seven

Title: Weekly Meeting

Date: 10/19/2020

Subject: RPI Menu Bar and VNC Server

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- Last week after I fixed BiasT, the following were the meeting objectives: - timestamps on the images differ in RPI and on the website. In one of them it is before the satellite passes and in the other it is after. - BiasT is still being monitored to see if any error occurs. For the past week it has been working fine. Currently making sure of wav files are at least 20 mbs.

- For the next step, Drew wanted me to work on the website to organize the images we get from the satellites. Each image corresponding to the satellite image will be in their section and instead of for daily storage, users will be able to see the last 7 days. - For the first task, I become familiar with the Workspress and Selenium. I have been getting familiar with the Wordpress interface as well as codes written with Selenium to control it. I plan to use Wordpress to build a website on my own in order to achieve this as well as implementing Selenium commands to see how they work. - To keep the 7 days old images instead of deleting them, I have to make a storage plan. In Wordpress currently there is 2 GB space. To keep just the daily images we use approximately 300 MB everyday. I will figure out a plan to decide how much space will be used to store the images.

Current Status

- Last week, the menu bar of RPI disappeared unexpectedly. I have run the following commands to get it back:

Remove lxpanel Config File Shell 1 2 3 4 $ sudo rm -r ~/.config/lxpanel $ startx

 or

$ sudo reboot

After making the reboot, it got stucked. (Website for the commands: https://www.kirkmakse.com/raspberry-pi-raspbian-missing-menu-bar/)

Now I get the error message “ VNC Server is not currently listening for Cloud connections”

- This week’s meeting is postponed to a later date this week. Waiting on Prof Jones to set the date. This week might be updated afterwards.

Accomplishments

- I got the credentials to connect to the Wordpress page of Kopernik and make changes.

Week Eight

Title: Weekly Meeting

Date: 10/26/2020

Subject: Bias T in New Session

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- I have run a test on Bias T to see how well it was working. When it is on schedule without any interruptions, it works like usual and we get clear images. A couple of days ago, we made a hard reboot to see if it would work. When I checked the resulting images, it was clear that Bias T was turned off. First thing I did was to turn on Bias T manually and run the grep… command again. I followed up the next day to see if it rebooted itself as scheduled → it did. Therefore, I focused on a little upgrade for crontab to rerun when RPI is rebooted.

Current Status

- I have updated the crontab file to:

1 0 * * * /home/pi/rtl_biast/build/src/rtl_biast -b 1 @reboot /home/pi/rtl_biast/build/src/start_application.sh 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot /home/pi/weather/predict/start_application.sh

Next steps:

First I will check if it runs like it used to on schedule. Then, I will make a hard reboot to see if the upgrade is working successfully.

Accomplishments

- Since x-term file is present in the new session, I have used the command:

Grep CRON /var/log/syslog

To once more to see how crontab reacts. Currently, it reboots itself everyday at midnight and we are getting clear images.

Week Nine

Title: Weekly Meeting

Date: 11/02/2020

Subject: FTP Server

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- Following up Bias T: the lines I added turned on the Bias T whenever there is a hard reboot. I followed the images for a week. Unfortunately, the results were inconsistent. At some hours of the day the images were clear, at some hour they weren’t. There were even times that at the same hour some images were clear and some were not. That is extremely strange because I have made the necessary tests to see if the update I have made to the crontab corrupts its usual process. The lines I added only get runned when there is a hard reboot. Thus, as expected it had not corrupted any usual activity. After Prof. Jones and IT technician Dave updated the whole system, now the Bias T is in its previous form where it cannot work with a hard reboot without a manual turning on Bias T.

- Website: Right now images go through Selenium and Wordpress in order to get posted on the website. As we have been discussing through last weeks, instead of Selenium and Wordpress, FTP server seems to be a better choice to get the images posted.

Current Status

- I have access to the Control Panel of FTP server. Andrew made a new account for me to reach the files where images are stored. The problem with that is with the newly made account, I could only reach a limited number of files.

- In order to use FTP server, I became accustomed to Filezilla and Cyberduck. I have been getting familiar with the interfaces and features as well as making the configuration.

- Next Requirements: Timeline of the images do not match with satellite passings and in some cases there are 3-4 copies of the same image in the folder. FTP server needs to work with NextGen Gallery to store the last 7 days of the satellite images.

- I am currently working on using Cyberduck to get a file manually to the gallery and then I will repeat the same process with a simple Python code to see the implementation. I need to get access to the path where images go. Right now I only have limited access to some of the files.

Accomplishments

- In my attempt to bring back the missing menu bar, I have updated the software of RPI. At that time, it was not enough to bring back the missing menu bar. After Dave made other adjustments, they realized that some necessary update of the system was missing. Thus, completing that now we have the menu bar back. - I got the basics of FTP account and interface. I have started to use the Cyberduck.

Week Ten

Title: Weekly Meeting

Date: 11/09/2020

Subject: FTP Server

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- Following up Bias T: After the whole system reboot, for a while Bias T was in its form where I have not added any @reboot lines. Then instead of making two @reboot lines, we just added one. Because it is actually only one of the crontab (root) that does the whole work. So now Bias T is:

1 0 * * * /home/pi/rtl_biast/build/src/rtl_biast -b 1 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot /home/pi/rtl_biast/build/src/rtl_biast -b 1

Crontab is working just fine now. My main focus right now is FTP server.

Current Status

- The path of the new FTP server account has changed to /public_html/wp-content/gallery/new and I have access to all the files.

The new program will take place of wp-bot which uses Selenium and Wordpress. When a simple version of this new program is completed, I will use this path to upload an image for testing purposes.

- I proposed to rearrange satellite files so that all I need to do is take them from the corresponding files. This would organize things in both RPI and website. Prof. Jones prefers me to check up on the images in the python program before I put them to corresponding places in the website. Because making an arrangement like I proposed would mean changing multiple program files. We will make a quick fix first.

Accomplishments

- Crontab is fixed and running. The update is also working just fine. I have been tracking crontab’s logs for a month now. FTP server is my main focus from now on.

Week Eleven

Title: Weekly Meeting

Date: 11/16/2020

Subject: FTP Server

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- In order to use a program to upload an image to server, Prof. Jones first suggested that I make changes in wp-bot. However, I could not find it in its folder for a while. Now it is present but we decided to make a new Python program that would replace wp-bot instead.

Current Status

- During uploading an image manually using Cyberduck, it warned me about an unsecured FTP connection. Thus, after consulting with Prof. Jones and Mr. Deskur I changed it to FTP-SSL.  

- Next step is to install FTP client or both server and client to RPI using the link Mr. Deskur shared with me.

Accomplishments

- I configured Cyberduck and manually uploaded an image to FTP server.

Week Twelve

Title: Weekly Meeting

Date: 11/23/2020

Subject: FTP Server

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- Reporting the updates on using FTP server instead of Selenium and Wordpress. I am currently using default library ftplib in Python to put FTP server in use. Since term presentation is in the coming weeks, this week Mr. Deskur will show me courtesy to meet twice if possible. - For Bias T, someone mysterious keeps messing with it. After I made it work, I updated it so that it would reboot itself even after a hard reboot.

Crontab after my update was:

1 0 * * * /home/pi/rtl_biast/build/src/rtl_biast -b 1 @reboot /home/pi/rtl_biast/build/src/start_application.sh 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot /home/pi/weather/predict/start_application.sh

Start_application.sh is basically a built-in script that when called, runs the programs at reboot/restart. After I fix crontab, it would run at midnight everyday. However, if you make a reboot during the day, the rest of the day you would get noisy images because crontab would not get rebooted till midnight.

Later on instead of using two reboot lines we turned it into:

1 0 * * * /home/pi/rtl_biast/build/src/rtl_biast -b 1 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot /home/pi/rtl_biast/build/src/rtl_biast -b 1

Someone keeps changing it to:

1 0 * * * (cd /home/pi/rtl_biast/build/src; ./rtl_biast -b 1) 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot (cd /home/pi/rtl_biast/build/src; ./rtl_biast -b 1)

Which is definitely not something that I have done.

Current Status

- I have coded a simple Python program just to connect to FTP server. Since there are built-in libraries in Python, it seems fairly manageable. First I will be implementing the program just for one satellite and then all four. The goal is to archive 7 days old images, not just that days’ images. - RPI did not have FTP server configurations. In order to have remote-access I followed the steps in this link: https://www.raspberrypi.org/documentation/remote-access/ftp.md#:~:text=FTP%20FTP%20%28File%20Transfer%20Protocol%29%20can%20be%20used,of%20the%20limited%20users%20is%20also%20required%20often

I created necessary groups and users with no permissions as well as the directories. I set up a password and made configurations. After all that, I restarted the system by sudo service pure-ftpd restart

The system did not freeze like the last time I tried to make a reboot. In that sense, things are on track right now.

Accomplishments

- FTP server for RPI (Pure-ftpd) is installed and configured. Cyberduck is also working. I could upload images manually as I tested it the last time. Next step is to

When I opened up my Wiki page to write down Week 12, I realized that my work before Week 9 was gone from the page as well as my electronic notes. Right now I am taking photos of every entry I am making to make sure something like this does not happen again and if so I will have the exact same entries.

Week Thirteen

Title: Weekly Meeting

Date: 11/30/2020

Subject: FTP Server

Attendance: Ezgi Cakir, Prof. Merwyn Jones, Andrew Deskur

Meeting Agenda

- Discussing the latest updates on the course of the project. Keeping an eye on Bias T, while working on FTP server. This week mostly has been the overview of the semester.

I changed crontab from:

1 0 * * * (cd /home/pi/rtl_biast/build/src; ./rtl_biast -b 1) 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot (cd /home/pi/rtl_biast/build/src; ./rtl_biast -b 1)

To:

1 0 * * * /home/pi/rtl_biast/build/src/rtl_biast -b 1 2 0 * * * /home/pi/weather/predict/schedule_all.sh @reboot /home/pi/rtl_biast/build/src/rtl_biast -b 1

Which Prof. Jones later on over ruled and changed the first version.

Current Status

- FTP server is not ready yet to replace the Wordpress and Selenium. I have been coding a very simple Python program just to make the simplest FTP server connection. IT seems that I would need more time to achieve this (setback).

Accomplishments

- The system is working fine without any errors which is an improvement since the first time I had taken over the project. There are multiple improvements that could be made. However, Bias T is stable and reboots itself even after a hard reboot. The system is fully updated. Menu bar is back. All system files are in place.