Kopernik - Bayar, Ilke; Demir, Cagkan

Week 1

Date: 09/26/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- Team members had meeting with Professor Jones and Andrew Deskur.

- Received information about the project.

- Introduced to necessary tools such as Selenium and Raspberry Pi.

- VNC Viewer installed to our PCs and remote access completed.

- Current coding files analysed and roadmap discussed.

- Company website analysed and problems founded about the uploading files to the archive section.

- https://www.n2yo.com/  --  this website is used to determine when a satellite will give us images in the today folder

Problems:

- No problems.

To-Do:

- Roadmap will be discussed with Prof. Jones and Andrew Deskur.

- To have information about how the current source code detects which satellite give those pictures.

Week 2

Date: 10/03/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- We had the second meeting with Drew and Prof. Jones. Our meeting started off with Prof. Jones’ expectations about the improvement of the wiki and our progress. Moreover, we went to the details again about the source code and what it does.

- We cleared off the existing files for NOAA15 and NOAA19 archives. Last time they were put on that page was late April and early May so they needed to clear off. Also, the photographs on NOAA 19’s archive were inaccurate since they included the other NOAA’s photographs on it.

- For running the source code, first we run the existing source code to check out what is happening. The first thing the source code did was clearing off the existing satellite photographs that were on the page. After we realized that it was unnecessary to remove the existing satellite photographs, we checked out the source code to find out which function removed those files in the first place. After we found out which function removed the existing satellite photographs, we made it a command line to get rid of that function for now.

- We already had the last-7-days archive section on the website for showing off the archive files on the website. Although that was nice to have, we needed more sections on the website to accomplish not only showing the last 7 days’ files, but we also needed the last 15 days section and the last 30 days section. For this situation, this week we created the last 15 days section on the Selenium for all NOAA15, NOAA18 and NOAA19 satellites.

- After we did run through the source code and created the last 15 days section for all satellites, we modified the source code to publish the satellite images in the archive section. To accomplish this, we wrote a new function to copy the updated satellite images not only to the Kopernik Weather Satellite Images but 15-day archive sections on Selenium. That was a good accomplishment to have to improve our progress to our main goal.

- After we implemented our new function to the source code, when we ran the source code, we realized that the time sleep periods were too long for this source code, so we shortened the time sleep duration to run the source code.

Problems:

- To remove the satellite images at the correct timing when the archive file is full (like 7 days or 15 days) we need to figure out a function to find out when to delete the existing satellite images on time.

To-Do:

- We will create the 30 days archive files for each satellite. Then, we will implement a new function on the source code for the 30-day archive file, same as the 15-day archive file.

- We have to implement a function to remove the satellite images at the correct timing when the archive file is full. we need to figure out a function to find out when to delete the existing satellite images on time.

- In order to run the source code to update the website, we have to rearrange the time sleep duration on the source code all over.

- In the next meeting, we will have to discuss when the Selenium is going to work and when the computer gets the new images from the satellites.

Week 3

Date: 10/10/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- We gathered for the third time for our weekly meeting with Drew and Prof. Jones. Unlike the last meeting, this week we received positive feedback from both ends. Ilke suggested an algorithm to get the pictures to the archive files but it got rejected by Prof. Jones since it was brute force coding. Furthermore, we learned that we do not need 15-day and 30-day archive files, just 7-day archive files will be sufficient for this project.

- One of the ideas we thought of on our brainstorming was source code would create a text file and keep tracking the amount of images coming on the same day. But unfortunately, we terminated that idea because it was taking too long to read and write up the text file again and again. So we need to change our the way we approach this situation.

- To make the 7-day archive directory correctly, first we needed to find the oldest images on the archive file to remove it when it expired. So, we managed to get the dates as string. But the problem was month was coming up as a word but what we needed was to get it as a number (for example Oct instead of 10). In order to fix this problem, we created a new library for translating words to integers.

- After we successfully transformed words to numbers on the files, we searched for a library to make the process efficient. We came up with the datetime library to compare the current date with the dates on the file at the archive to pull the oldest image. 

- Some of the archive files such as NOAA19ARCHIVE, NOAA18ARCHIVE, NOAA15ARCHIVE had a limit at the 21th file. This feature was both unnecessary and unhelpful for our project. We found out which line of code was causing this issue and we removed that feature from the source code.

- After making these edits, the source code now removes the expired images on the 7-day archive file.

- Typos on the website fixed for the archives. For example, it was shown as NOAA15 - Archive on the NOAA18 Archive section, we fixed that.

Problems:

- The source code is making some steps unnecessarily and it is making the process slower than it should be.

To-Do:

- Next week, we are going to figure out how to reduce the runtime of the source code and make it more efficiently.

- Tuesday is the 7th day of the source code working properly. We will check the website to see.

Week 4

Date: 10/17/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, because we successfully completed all of the objectives for the project, we have given additional tasks about the archives. Prof. Jones stated that the archive section of the website is working correctly but it might be too confusing for viewers because it has got a lot of pictures and the dates were unclear. 

- After this suggestion, Cagkan stated that an archive is supposed to be confusing since the meaning of the “archive” is gathering data from previous days. Prof. Jones insisted that we have to design the archive section of the website in a much clear way.

- After this suggestion, we started brainstorming about how we can design the archive section so that website visitors can see the pictures more clearly. In the meeting, we came up with an idea that we will make 7 sections in the archive folders, so visitors can see the archive files day-by-day. After this suggestion was finally accepted by everybody in attendance, we concluded our weekly meeting and started our work.

- In order to achieve this objective Ilke and Cagkan had a meeting to figure out the best way to make the archive folder with a day-by-day section. We looked at the source code, checked out the website and spent time on Wordpress.

- Ilke came up with an idea of creating 7 archive files and after each day passes, the pictures are going to be sent in their respective folders. Although this strategy, this way of thinking did not got approved by Prof. Jones, this is the solution we came up with.

- To accomplish our objective, we created 7 additional folders named NOAA15D1, NOAA15D2 and all the way to the NOAA15D7 on Wordpress. Then we improved our source code with these folders.

Problems:

- While we were closing on finishing our objective, we encountered a VNC Viewer error and it denied our connection to the Raspberry Pi. “VNC Server is not currently listening for Cloud connections.”

- Now we have got too many folders to deal with, working on source code is a lot more difficult now. Note that Prof. Jones predicted that result on the last meeting.

To-Do:

- We need to find a more efficient way to make the archive section based on a day-by-day view.

Week 5

Date: 10/24/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, we kept our focus on the archive section of our website. For the viewers it was too confusing to check out the photos since it has a lot of pictures and the dates were unclear.

- Since we brainstormed but could not find a way for showing it day-by-day basis last week, this week we changed our way of approach and tried to show it what we called “pass-by-pass basis” which can be described as putting the photos took on each satellite pass on the very same row. Note that this way of approach already mentioned by Prof. Jones in the last meeting.

- After this work of thought, we tried numerous ways to show the photos on the archive to be 7 photos on each row. First, Cagkan reduced the width of the photos from 120 to 90 to fit 7 photos on each row. While the width of the photos got smaller, 7 photos on a row did not fit properly.

- Then we looked out for a better way to make it a fit. We looked for source codes on Wordpress, tried to reach out for size settings on Wordpress and gave it some try. Then Ilke found the correct setting to fit 7 photos in a row, and he made the width of the photos section of the website 1110. Note that the width was 860 before.

- In order to reach out the setting for photo section width, Ilke followed settings, appearance, team editor, style.css, nextgen gallery position path and made the changes there. After doing that, now we can see 7 photos in a row in our archive sections and it is a pretty good way to start making archive sections easier to look into.

- Also we changed the view of the photos to “caption view”. As it is seen and discussed in the last meeting, with this type of view, we can get additional information about each photo.

Problems:

- We tried to figure out a way of changing captions automatically but could not figure it out. Instead, we changed 10-20 pictures’ captions on NOAA15 Archive to show everyone on the team how it is going to look if we changed all of the captions. 

- Moreover, we still do not know where the captions are coming from and how to view them on the computer before it is released on Wordpress.

To-Do:

- We are going to have a team discussion in our next meeting about the captions with all details. How to view it on file and how to run a code to copy the title and paste it on description.

Week 6

Date: 10/31/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, Cagkan and Ilke explained their accomplishments about how they fit 7 photos in one row on archive sections by changing the photo sections width and changing the view settings as “caption view”.

- Later on, even though Drew and Prof. Jones impressed the way the archive folders looking with 7 photos on each row, they expressed that the archive folders have to be clearer and better looking. Then, we concluded our meeting by pointing that: we need to have better captions, we need to change the sorting (sort by newest, instead of sort by oldest).

- After the meeting, we started our weekly work with writing a function that edits the caption properly. Since the caption is coming from the description part of the photos, we implemented a function that edits description with the “title” and “type”. Note that photos coming with “Pristine” does not have any “type” on the description, so each photo with null “type”, our function adds “Pristine” on their caption. 

- Moreover, we changed the sorting to be shown as newest photo first. It was showing the oldest photos on top before we made this innovation. Since both Drew and Prof. Jones addressed that newer photos on top would be much better, we edited the sorting section like that. Note that we had a problem with deletion of the photos since the delete function works based on oldest photos sorted on top. We fixed that problem by making the sorting with oldest on top when the program needs to clear old photos, then turning it back to newest on top when the deletion is successfully completed, automatically.

- Last but not least, we also implemented these innovations the weather section of our website along with the archive section.

Problems:

- No problems.

To-Do:

- We are going to implement a function to our project to keep a log about satellite passes and website updates. Based on Prof. Jones’ recommendations, what we are planning to do about the forthcoming log system is creating a .txt file and implement a program to fill out the .txt on each website update.

- Also, as discussed on last week’s meeting, we are going to clear the archive sections all together to give our archive sections on the website a clean start. Part of the reason why we are going to do this is that some of the satellite photos came out faulty when we are implementing our functions during our project.

Week 7

Date: 11/07/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- We cancelled our weekly meeting because it was Halloween. We talked with Prof. Jones via email to what we are going to do this week. We concluded that we solely are going to do what is on our to-do list on Week 6. 

- So, we started our to-do list on Week 6 which is implementing a function to our project to keep a log about satellite passes and website updates and clearing the archive sections all together to give our archive sections on the website a clean start.

- We started off our weekly work for with clearing the archive folders to give the website a clean start. In order to clear the archive folders, we headed to the NextGen Gallery, clicked on Manage Galleries and deleted pictures on WordPress. 

- Moreover, next step is to create a log file to keep a log about satellite passes and website updates. For this implementation, first we created log.txt file in weather/today folder. We wrote a function on wp_bot.py to write each log to the log.txt.

- On Saturday evening, while we are looking on the website, we noticed that the code did not run automatically. We realized that Raspberry Pi froze sometimes this week. This is happened because the computer could not reach the website. We believe that this could happen one or two ways: no internet connection while the code tries to run or getting an error from the server. When we checked the backlog, we saw that the code did not run properly each time when satellite passes and that happened 4 times on total between 3 Nov and 6 Nov. After failing for 4 times on total, the RAM reached its maximum capacity with these requests and the Raspberry Pi got overloaded. We closed the Python files that still trying to run and ran the code manually. After that we did not meet with any problems.

Problems:

- No problems on creating a log file and clearing the archives.

- We encounter some problems about the connection between WordPress and Raspberry Pi.

To-Do:

- In the next meeting, we will discuss with Drew and Prof. Jones about the connection problems happened between WordPress and Raspberry Pi this week.

- Moreover, we will discuss with Drew and Prof. Jones about the WordPress updates.

Week 8

Date: 11/14/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, Cagkan and Ilke explained their accomplishments about how they made a log file and built a program to add log information each time the website updated.

- Later on, for the most of our time in our meeting, we discussed about the errors we received about how the connection between WordPress and Raspberry Pi failed 4 times in a row to freeze the Raspberry Pi completely. Each one of in the attendance shared their thoughts and understanding about the problem and we concluded the meeting with everyone is going to check out the main reason of the problem and how to fix it.

- Meanwhile, thanks to Drew, we finally found out to see the texts on our pictures. Since we could not open up with Text Editor and reach out the texts from the image, by opening up via Notepad, now we can see the whole metadata with descriptions.

- But unfortunately, the Text Editor on Linux still cannot open the pictures with its build-in Text Editor. New application needed to see it on Linux as well. But we believe this accomplishment is enough for this topic since opening them up with Notepad proved that the description information is in the pictures.

- Furthermore, we made our research about why we got those errors on the last week. So, we concluded that the problem was coming from the timesleep. The time we have given to our program was too short to make the uploads successfully, so the timesleep timer was too short. To fix the problem, we increased the timesleep to 15. Note that it was 5 before we made this edit.

Problems:

- Raspberry Pi still working slowly. We are researching about why it is working that slow. 

To-Do:

- We will give our focus to the issue of working slowly. We will make some research about why it is working that slow and how can we improve the speed of both the program and the Raspberry Pi.

Week 9

Date: 11/21/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, Cagkan and Ilke opened up the discussion about the problems that occur when the code tried to run itself. The problem was the Raspberry Pi was hanging when the program tried to run itself. In order to find out was caused the problem, Prof. Jones suggested two things: first, we should update the log each time when the code ran (successfully or unsuccessfully) and go back to the original code and add our new features one by one to see that which feature was causing the problem.

- First, we only ran the weather file update section of the code, which was the simplest version of our program. We ran the code like this for 48 hours. No problems occurred on this version of the code.

- After that, we implemented the log writer function to this basic code and ran for 48 hours. Unlike the first one, we ran into some problems, which is Raspberry Pi hanged again.

- After we ran into the problem, we understand that the log writer function caused the problem. To make sure of that, we implemented all of our features again except the log writer function and ran the code for 48 hours. No problems occurred on this version of the code.

- We realized that sometimes internet speed was too slow, so image uploads did not complete successfully. In order to fix this problem, we increased the timesleep of the upload images function.

Problems:

- When the log writer function implemented, the function needs to reach log.txt while running. When we manually ran the code, the log writer function can reach the log.txt but it cannot reach log.txt when the code ran itself.

To-Do:

- We will discuss with Prof. Jones about our log writer function to reach log.txt file in today folder when it ran automatically.

- Because of the timesleep numbers need to match with the internet speed, we increased the timesleep. This was our solution for this problem, we will discuss with Prof. Jones and Drew about this situation.

Week 10

Date: 11/28/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, all of our team members addressed the situation on the log writer function. When we implemented the function to the source code, the source code did not run automatically. Prof. Jones told Ilke and Cagkan to fix that function and implement it to our program. Furthermore, both Drew and Prof. Jones pointed out the descriptions of the images in the website. They told us get rid of “enhancement” on every description, remove “sensor 4”, type “thermal IR” instead of “thermal infrared” and type “sea temp” instead of sea temperature. Furthermore, Prof. Jones also asked us to make improvements on the source code’s design.

- First, we founded out the error that caused log writer function fail. It was a directory fault, meaning the function could not find the logs. We fixed that by adding the directory path to the function. After making this improvement, the function ran without any problem.

- After that, we moved on to the description editing part of our weekly work. We took notes on every suggestion for the descriptions and made these edits. To be exact, we got rid of the “enhancement” part on every description, removed “sensor 4”, wrote “thermal IR” instead of “thermal infrared” and wrote “sea temp” instead of sea temperature. 

- Furthermore, we started working on the design part of our source code. We removed unused and excess functions and made our source code simpler.

Problems:

- Even though we made almost all of the edits that suggested on the descriptions, satellite pictures coming with type of HVCT sometimes comes without any descriptions and that makes the caption faulty.

To-Do:

- We will discuss with Prof. Jones and Drew about the HVCT type satellite pictures that comes without descriptions.

Week 11

Date: 12/5/2022

Attendance: Andrew Deskur, Cagkan Demir, Ilke Bayar, Prof. Merwyn Jones

---

Accomplishments:

- In the last meeting, we talked through our accomplishments that we did last week. We explained that how we found and fixed the log writer function and how we updated the descriptions for each picture, making adjustments on description texts. Last but not least, we explained how we made the project source code simpler. After the presenting this week’s accomplishments, we moved on to our goals for this week. Since we added many features for all 10 weeks, Prof. Jones stated that the source code might be too difficult to read through for developers, and he asked us to add command lines. At last, he also asked us to upgrade the log writer function to log each time the sorting function worked since we got program fail last week on sorting.

- First, we started off with adding command lines around or source code. We added explanation and information on each part of the source code that we felt necessary.

- After that, we moved on to the log writer function upgrade. Since we had a problem with sorting part of the program for one time, Prof. Jones suggested that the log writer function should log the sorting function as well. We wrote that implementation to another .py file and it is ready for implementation. 

- Additionally, we also made a simpler version on another .py file.

Problems:

- Even though we think that both simpler version of the source code and the log writer function is mostly ready for implementation, we are not sure that big of a change just before semester ends is a good idea.

To-Do:

- We will discuss with Prof. Jones and Drew about the implementation of upgraded log writer function and simpler version of the source code to our project.