View source for Kopernik - Emir Tasan; Mert Bekirogullari; Taha Ucar; Teoman Kaman;

''<big><big>Week 1 09/10/2024</big></big>''
--------------------------------------------------------------------
<br>
'''Summary:''' 
This week, our goal was to successfully access the Raspberry Pi and demonstrate specific tasks related to file management and system understanding.

'''Accomplishments:'''

<br>
# Group meeting has been done.


'''To-Do:'''
Successfully logged into the Raspberry Pi, demonstrating the ability to establish a connection.
Understood the process of connecting to the Raspberry Pi and related it to the instructional link provided.
Identified the folders containing the 'pi' folder within the Raspberry Pi directory structure.
Demonstrated the ability to copy and move files from 'root/home/pi' to the new system.
<br>
# New images will be viewed to see if the crontab issue has been resolved.
<br>
''<big><big>Week 2 09/12/2024</big></big>''
--------------------------------------------------------------------
<br>
'''Summary:'''
This week focused on advancing the Kopernik project, specifically tasks related to the "pi" folder. The group conducted a detailed meeting to discuss the week's objectives, primarily centering on working with the system scripts. The key activities included exploring the system to locate the "schedule_all.sh" script, attempting to execute it, and investigating the integration and access of "wp_bot_sav.py." Despite facing challenges with the script's execution, valuable progress was made in familiarizing with the system and identifying potential issues for further resolution.

<br>
'''Accomplishments:'''
<br>
# Group meeting has been done.
Conducted a group meeting to discuss the tasks for the week and to align on the approach for working with the scripts.
Explored the system to locate the "pi" folder and attempted to find the "schedule_all.sh" script.
Attempted to run the "schedule_all.sh" script to ensure proper execution.
Investigated how to access "wp_bot_sav.py" as part of the tasks.
Although initial attempts to run the script(schedule_all.sh and schedule_satellite.sh) were unsuccessful(no directory found error in both scripts), progress was made in understanding the system and identifying potential issues.

Found the no directory found issue in the scripts. It was because we put the pi folder into new system so in the script the directories should be fixed.


''<big><big>Week 3 09/20/2024</big></big>''
--------------------------------------------------------------------
<br>
Summary: 
    - Conducted research on Gpredict vs Predict, focusing on platform differences, user interfaces, and satellite tracking capabilities.
    - Tested using schedule_all.sh with Gpredict, noting that the GUI is required for certain operations when working over SSH.
    - Continued exploring Selenium for task automation and integration with other tools.

Accomplishments: 
    - Completed detailed comparison research between Gpredict and Predict for satellite tracking needs.
    - Successfully executed schedule_all.sh with Gpredict, identified that GUI support is required for full functionality.
 
To-Do:
    - Investigate methods for running Gpredict via SSH with screen sharing to bypass GUI limitations.(optional)
    - Develop on wx2 image file processing scripts, focusing on optimizing file handling and data output. Get familirazed with the image processing scripts
    - Finalize Selenium automation scripts for further testing and integration in the upcoming week.

<be>

''<big><big>Week 4 10/14/2024</big></big>''
--------------------------------------------------------------------
'''Summary:'''

 -This week, we debugged and refined key scripts involved in automating the processing and uploading of satellite weather data. After testing, the receive_and_process_satellite.sh script successfully captured and processed satellite data, and updates were made to improve file management and web integration.

'''Accomplishments:'''

  '''Updated receive_and_process_satellite.sh:''' Optimized satellite data capture and image generation, ensuring wxtoimg functions correctly.

 '''Refined dir_script.sh:''' Improved the execution of file cleanup, image prep, and data uploads by managing subprocesses efficiently.

 '''Enhanced prep_web_images.py:''' Debugged the image replacement process and added file size checks to avoid processing incomplete files.

 '''Modified remove_old_files.py:''' Ensured proper deletion of files older than 7 days with added error notifications.

'''Downloaded and got access to the new vnc tool:''' Got the permisson to access to the NOAA system for the new vnc tool VNC Viewer, got familirazed with the new vnc tool

'''To-Do:'''

 -Continue testing for stability and performance.
 -Optimize satellite data capture in receive_and_process_satellite.sh.
 -Document script changes for future reference.
 -Implement automizatiom for the receive_and_process.sh script to automatically do the job

''<big><big>Week 5 10/21/2024</big></big>''
--------------------------------------------------------------------
'''Summary:'''
This week, we updated the satellite processing script (receive_and_process_satellite.sh) to dynamically fetch satellite data from the TLE file. We improved error handling, automated the retrieval of satellite frequencies, and fixed file path issues. The script now automatically processes captured WAV files and generates various satellite images using wxmap and wxtoimg tools. However, we encountered issues with wxtoimg and wxmap not functioning properly, despite the libraries being installed. Additionally, we faced permission issues when trying to install a GUI-related library due to password restrictions.

'''Accomplishments:'''
 -Added dynamic satellite data retrieval from weather.tle.
 -Automated frequency assignment for NOAA 15, 18, 19 satellites.
 -Improved file handling by removing spaces from satellite names.
 -Fixed paths for processing binaries (wxmap and wxtoimg).
 -Improved error handling for missing data or failed file creation.
 -Created the .wav file from receive_and_process file.
 -Investigated issues with wxtoimg and wxmap not working properly, despite the libraries being installed.

'''To-Do:'''

 -Resolve issues with wxtoimg and wxmap not functioning correctly after installation.
 -Investigate the permission issue related to installing the GUI library (password required for installation couldn't installed).
 -fix the bugs inside dir_script scripts 
 -Expand the satellite list to include more satellites if needed.
 -It doesn't take the data from next event even though it creates a .wav file will fix the issue
 -Implement more robust logging for debugging and error tracking.

''<big><big>Week 6 10/28/2024</big></big>''
--------------------------------------------------------------------
'''Summary:'''  
This week, we focused on refining the satellite tracking and data processing system, automating data collection and improving the quality of output images from satellite signal recordings.

'''Accomplishments:'''

1.⁠ ⁠'''Satellite Pass Calculation Using Gpredict and Python Integration:'''  
   Configured the ⁠ receive_and_process_satellite.sh ⁠ script to pull TLE (Two-Line Element) data directly from Gpredict. This data is processed with a Python script that calculates the next satellite pass time for our specific location, using the ⁠ ephem ⁠ library. Based on the calculated pass, the system records a 10-minute segment of satellite data and converts it to a WAV file.

'''2.⁠ ⁠Troubleshooting RTL-SDR Errors:'''  
   Encountered persistent issues, specifically ⁠ usb_claim_interface error -6 ⁠ and ⁠ PLL not locked ⁠ errors, during the WAV file creation process. These issues appear to stem from conflicting processes attempting to access the RTL-SDR device or failures in device locking. Attempted to troubleshoot using methods like terminating conflicting processes with ⁠ sudo pkill ⁠, though this was only partially effective.

'''3.⁠ ⁠SatDump Configuration and NOAA Data Pipeline:'''  
   Configured SatDump to decode NOAA satellite data from the generated WAV files. Tested various sample rates and input formats, yielding images of acceptable value but with room for quality improvement. Currently, image quality needs enhancement, and there’s a need to integrate this process more closely with other scripts to transition from a manual to an automated workflow.

'''*To-Do:*'''

  ⁠-Continue troubleshooting ⁠ usb_claim_interface ⁠ and ⁠ PLL not locked ⁠ errors to stabilize device access.
  -⁠Improve image quality produced by the NOAA data pipeline by fine-tuning SatDump settings and sample rates.
  -⁠Work on linking and automating the current scripts to create a fully integrated, hands-free satellite data processing system.
  -Set up crontab jobs to automatically run the scripts at specific times, ensuring consistent data capture without manual intervention.

''<big><big>Week 7 11/04/2024</big></big>''
--------------------------------------------------------------------

'''Summary:''' Continued development and testing of satellite tracking and data processing scripts. Added functionality for image generation and troubleshooting connection errors.

'''Accomplishments:'''

  -Developed a new shell script, satdump_process.sh, which retrieves the latest .wav file from a specified directory and processes it into an image using satdumpi.

  -Integrated generate_images.py to create seven different types of images (e.g., infrared, sea level, mcir, etc.) from the processed image file.

  -Attempted to use wp_bot for automating image uploads to the website, but encountered errors that prevented successful uploads. Troubleshooting on wp_bot is ongoing.

  -Programmed satellite tracking to capture passes between 20 and 90 degrees. Addressed recurring USB connection error -6 by updating the script to restart automatically upon detecting this error.

  -Implemented a sudo kill command to terminate other processes using RTL libraries, ensuring priority access for our scripts.

  -Created a test_receive_and_process_satellite.sh script for testing receive_and_process_satellite.sh functions in a shorter time frame. Testing revealed issues, so the test script is under further revision.

'''To-Do:'''

  -Resolve wp_bot issues to enable automated image uploads.

  -Finalize and optimize the test_receive_and_process_satellite.sh script for more reliable testing.

  -Continue addressing USB connection stability and error handling.

  -Refine satdump_process.sh and generate_images.py scripts to ensure high-quality image outputs for all image types.

''<big><big>Week 8 11/11/2024</big></big>''
--------------------------------------------------------------------

'''Summary:'''
  -This week, the team focused on improving the satellite tracking setup, addressing key technical issues, and exploring alternative data sources for satellite pass predictions.

'''Accomplishments:'''

  -Doppler Frequency Adjustment: Added Doppler frequency adjustments to enhance signal quality.
  -USB Connection Fix: Resolved USB connection errors, though countdown calculations still need verification against Gpredict's data for consistency.
  -Image Processing Trials: Tested methods for image enhancement using OpenCV and Satdump, though quality improvements are ongoing.
  -Biast Installation: Successfully installed and configured biast to support signal capture.
  -Script Update and Web Scraping: Updated the receive and process script to create WAV files, though there are still accuracy issues in the output. To bypass calculation complexities, began web scraping a reliable data source for satellite pass details.
  -tried different approaches for generating 7 images(mcir,infrared, etc..) with satdump and opencv


'''To-Do:'''

  -Verify countdown calculations and refine image processing methods.
  -Finalize web scraping integration to streamline data acquisition for satellite tracking.
  -for further image processing research more (gdal, wxtoimg)


''<big><big>Week 9 11/18/2024</big></big>''
--------------------------------------------------------------------
'''Summary'''
This week, significant progress was made in satellite data processing and image generation. The system successfully generated and processed WAV files and visual outputs, focusing on improving quality and functionality.

---

'''Accomplishments'''

  - Receive and Process Satellite Script:
    - Successfully tested with NOAA 18, generating a valid WAV file in the `predict` directory.
    - Updated the script to process the latest three WAV files for better image outputs.

  - Image Processing:
    - WxToImg was used to generate Sea Surface Thermal (SST), Thermal, and ZA images with good quality.
    - OpenCV and Pillow were tested but didn't produce satisfactory results.
    - Satdump produced grayscale-only images, so the focus shifted to WxToImg for reliable outputs.

 - Web Scraping:
   - Successfully implemented web scraping for satellite data, replacing complex calculations.
   - The scraped data provided accurate results, validated by successful WAV file creation and image processing.

 - Biast Installation:
   - Installed and configured Biast to enhance system performance, leading to better reception and image generation results.

---

'''To-Do'''
 - Resolve issues with WxToImg for generating DHVCT and MCIR images.
 - Improve overall image quality and expand the range of usable image types.