Kopernik Pt.2 - Emir Tasan; Mert Bekirogullari; Taha Ucar; Mehmet Ozdemir; Teoman Kaman;

Week 1 02/18/2025

---

Summary

 Enhanced recording scripts, updated location/time settings, and automated data acquisition and scheduling. Explored alternative SDR solutions.

Accomplishments

Recording Scripts:  
 - Auto-creation of directories, termination of existing ⁠ rtl_fm ⁠ processes, absolute path usage, and detailed logging.

Script Updates: 
 - Updated ⁠ new.py ⁠ with corrected coordinates, timezone (EST), noise command, and PNG management.

SDR Integration:  
 - Connected SDR to gpredict, adjusted NOAA15 frequency, and tested gqrx as an alternative.

Data Pipeline:  
 - Created “Emir” folder, developed ⁠ n2yo.py ⁠ for data scraping, used ⁠ filter.py ⁠ to refine data, scheduled recordings with ⁠ schedule.py ⁠, and automated the workflow with ⁠ runall.py ⁠.

To Do

 -Test and debug record.py ⁠and automation loop.
 -Resolve remaining SDR and scheduling issues.
 -Fine-tune data processing and logging.

Week 2 02/25/2025

---

Summary

 This week, we focused on refining the satellite image processing pipeline, debugging existing scripts, and optimizing the scheduling system. We also improved the prediction and recording workflows to enhance data acquisition reliability.

Accomplishments

Recording & Processing:
 -Successfully tested and confirmed that record_passes generates valid WAV files.
 -Implemented and configured the predict software for our system, ensuring seamless execution.
 -Updated TLE files using schedule_all.sh and calculated satellite pass predictions.
 -Filtered valid satellite passes with an elevation range of 19 to 91 degrees using schedule_satellite.sh.
 -Successfully generated a WAV file with receive_and_process_satellite and processed it into an image.

Script Debugging & Enhancements:
 -Attempted to generate WAV files using the ephem library to predict satellite movement, but encountered issues and decided to halt further development.
 -Identified and resolved an issue in the script where wxotimg was not running in quiet mode, preventing popup interruptions.
 -Tested new WAV files and attempted fixes for mcir and hcvt, which were producing grayscale images instead of colored ones.
 -Confirmed that thermal and sea images generated correctly.

To Do

 -Investigate and resolve why mcir and hcvt images are in grayscale instead of their intended color formats.
 -Debug the missing map overlay in the generated images.
 -Continue optimizing the scheduling and recording processes for improved efficiency.

Week 3 03/04/2025

---

Summary

 This week, we focused on improving file organization, satellite data processing, and image management. We refined naming conventions, automated folder-based processing, and enhanced cleanup procedures. Additionally, we integrated new overlays and debugging efforts while ensuring periodic maintenance updates.

Accomplishments

  File Management & Processing:
 -Renamed the satellite to First Satellite for better file naming consistency.
 -Implemented a system to locate corresponding satellite folders and process the latest data in each folder.
 -Introduced an automatic cleanup mechanism that removes images older than 7 days.

 Image Enhancements & Debugging:
 -Added map overlay to the generated images.
 -Identified an issue with fog artifacts in some images that need further fixing.
 -Successfully made MCIR and MCIR-Precip images work.
 -Still need to work on HCVT processing.

 System & Automation Updates:
 -Integrated Teoman’s code into the project.
 -Updated files to automatically clean up every 10 days.
 -Updated the RTL command to improve functionality.
 -Reviewed the WP bot for performance and issues.
 -Adjusted satellite processing logic based on All Pass data.

To Do

 -Fix the fog issue in some images with the map overlay.
 -Continue working on making HCVT processing functional.
 -Further optimize satellite processing to improve efficiency.

Week 4 03/18/2025

---

Summary

-This week, we focused on improving satellite image quality, refining automation scripts, and troubleshooting map overlay issues. We enhanced noise reduction techniques, optimized processing efficiency, and explored alternative georeferencing solutions.

Accomplishments

Enhancements & Processing:

-Modified wp_bot for better functionality.
-Improved apply_enhancements.sh for noise reduction and overlay application.
-Enhanced file naming, logging, and cleanup mechanisms.

Image Processing & Debugging:

-Applied stronger noise reduction using ImageMagick.
-Addressed map overlay issues but faced compatibility problems with WXtoImg and Keplers-Updater.
-Switched to QGIS for georeferencing and noise reduction, improving image quality.

System & Automation Updates:

-Standardized overlay application across all processed images.
-Ensured better handling of temporary files to streamline workflows.

To Do

-Optimize script to process only new satellite files.
-Fine-tune noise reduction and overlay parameters for better image quality.

Week 5 03/25/2025

---

Summary

 This week, we focused on optimizing the image generation workflow by eliminating redundant processing across satellite folders. We also improved automation for image cleanup, updated the enhancement pipeline, and refined overlay and GIF display features. Significant progress was made in streamlining file management and testing web upload capabilities.

Accomplishments

File Management & Optimization:
 -Fixed the issue where all three NOAA folders were being processed redundantly.
 -Optimized the system to take the newest file from each individual folder instead of reprocessing all files across folders.
 -Cleaned up development files, test images, and unnecessary clutter to improve workspace efficiency.

Archiving & Automation:
 -Created a new script, move_images.sh, which is called after apply_enhancement.
 -Moves the newest images into the today folder.
 -Transfers older images into the archive_images folder.
 -Developed a script to automatically delete website images older than 7 days, ensuring up-to-date content on the web server.

Upload Testing:
 -Upload attempts were made to various NOAA satellite folders.
 -Upload to NOAA15 was successful.
 -Uploads to other folders failed and are currently under investigation.

File Order & Resizing:
 -Files are sorted to maintain a consistent frame order.
 -Images are resized using the high-quality LANCZOS filter.

Date/Time & Imagetype Extraction:
 -A regular expression is used to extract the 8-digit date and 6-digit time from the filename.
 -The imagetype is extracted as the second element in the filename, based on underscores.

Overlay Improvements:
 Overlays now include:
 -Satellite name (centered)
 -Time (top-right)
 -Imagetype (top-left)
 -Date (below the time)
 -Deprecated textsize() was replaced with textbbox() for more accurate text positioning.

GIF Output & Display:
 -GIFs are named using the format: satellite-yyyymmdd-hhmmss.gif.
 -Images are converted to ‘P’ mode with an adaptive palette for optimal GIF quality.
 -A Tkinter window displays the GIF on HDMI‑2 using the geometry setting "1920x1080+2560+0".
 -Escape key closes the window.
 -Proper handling of KeyboardInterrupt is implemented.

Cleanup:
 -Old GIFs are automatically deleted after the retention period to maintain storage efficiency.

To Do

 -Investigate and resolve upload failures for folders other than NOAA15.
 -Continue testing automated website cleanup and folder updating.
 -Monitor overlay formatting and display output for errors.
 -Refine archiving and file movement scripts if needed.

Week 6 04/01/2025

---

Summary

 This week, we improved the automated image display workflow by modifying the GIF generation process to ensure consistent behavior on the second monitor. Additional debugging was performed on hardware connectivity issues, and web upload functionality was enhanced with new metadata automation for the weather gallery.

Accomplishments:

GIF Display & Automation:
-Updated the gif_akb.py script to always display GIFs on the second monitor.
-The apply_enhancements script now calls gif_akb.py after move_images.sh is executed.
-Ensures that once a new image is detected and moved to the today folder, a new GIF is generated and displayed automatically.
-Old GIFs are automatically deleted before the new one is shown.

Hardware Debugging:
-Encountered a USB claim interface error -6 with the RTL2838 dongle.
-Used lsusb and lsusb -t to identify the device at port 1-1.4.
-Resolved the issue by unbinding the device using: sudo sh -c ‘echo “1-1.4” > /sys/bus/usb/drivers/usb/unbind’

Web Upload & Metadata Automation:
-Uploaded images to the weather folder successfully.
-Investigated the bias-t power option for further compatibility research.
-Updated wp_bot_upload with 5 new functions to automate gallery metadata:
-open_manage_galleries: Navigates to the Manage Galleries page.
-select_gal: Finds and clicks the edit link for the “weather” gallery.
-parse_filename: Extracts date, time, NOAA number, and sensor type from filenames.
-enter_text: Iterates through image rows and applies metadata to the ALT/Title and Description fields.
-click_save_changes: Scrolls and clicks the Save Changes button using JavaScript.
-The main function was updated to call these new steps after image uploads for full automation.

To Do

-Continue testing image display for edge cases and stability
-Finalize bias-t power option testing
-Monitor dongle behavior for recurring USB errors
-Expand metadata automation to other gallery folders if needed

Week 7 04/08/2025

---

Summary

 This week focused on automating the deletion of old images from WordPress galleries, integrating the upload script into the enhancement pipeline, and continuing work on automating the GIF creation process. Debugging and cleanup efforts were also carried out to improve stability and storage efficiency.

Accomplishments

WordPress Gallery Cleanup Automation:
-Created the script wp_bot_delete.py to automate old image deletion from WordPress galleries.
-The script performs the following:
-Logs into the WordPress admin panel and navigates to the NextGEN Gallery section.
-Scans and opens specified galleries (NOAA15, NOAA18, NOAA19, weather).
-Extracts the date of each image from its metadata.
-Selects images older than 7 days for deletion.
-Uses bulk actions to delete selected images, confirms the alert, and saves changes.
-Iteratively processes all listed galleries, logs progress, and exits cleanly after execution.

Integration & Script Updates:
-Integrated wp_bot_upload.py into the apply_enhancement.sh pipeline.
-Encountered an error during integration and attempted to fix it by enabling headless mode.
-Added a code block to wp_bot_upload.py to support smoother automation.

GIF Automation (In Progress):
-Continued developing the automation of the GIF creation process through the script.
-Initial tests show partial success: the script sometimes works automatically, but occasionally raises errors.
-Ongoing debugging to identify the cause of inconsistent behavior.

General Cleanup:
-Deleted unnecessary files from the project directory to free up storage space and maintain system performance.

To Do

-Finalize debugging of the GIF automation script for consistent execution.
-Continue testing wp_bot_upload integration and address any remaining issues.
-Investigate intermittent failures in WordPress automation.
-Expand wp_bot_delete to support dynamic gallery input if needed.

Week 8 04/17/2025

---

Summary

-This week, the focus was on improving automation, organizing scripts for better workflow management, enhancing image sorting and deletion features in the WordPress gallery, and optimizing the execution time of the overall pipeline. Additional debugging and logging functionalities were also added to increase transparency and efficiency.

Accomplishments

WordPress Bot Improvements:
-Added new functions to the WordPress bot to ensure the weather gallery only contains the latest 7 uploaded images.
-Introduced metadata functions and improved the deletion process for images older than 7 days.
-Now checks every page of each gallery to identify and apply updates correctly.
-Created a sorting script to reorder gallery images, although some galleries currently fail to sort due to issues on the website side.
-Assembled all existing WordPress bot scripts into a single unified script named wp_bot_new for cleaner execution.
-Code performance is improved, making the bot more stable and responsive across operations.

Workflow Optimization & Automation:
-Without the recording process, the full pipeline now completes in 4–5 minutes, significantly improving performance.
-Organized the entire workflow using a script dictionary to run each process step-by-step in a structured manner.
-Added logging for each part of the process, with outputs saved into a logs folder, including a master log for overall tracking.
-Debugged previous error handling issues to ensure more stable execution across the pipeline.
-Fully automated the workflow process, from enhancements to upload and cleanup.

To Do

-Resolve sorting issues caused by website limitations.
-Test the new wp_bot_new script across more galleries for stability.
-Continue refining log formatting and error reporting for better readability.
-Monitor full automation for potential edge-case errors or performance drops.

Week 9 04/22/2025

---

Summary

-This week, we focused on improving metadata generation, gallery image sorting, and overall workflow documentation. Additional efforts were made to enhance system reliability, error handling, and RTL functionality. The improvements ensure more stable execution and better organization across the automated processes.

Accomplishments

Metadata & File Handling:
-Updated the metadata function to support a new filename format.
-Ensures that all uploaded images in the weather gallery receive accurate metadata regardless of naming variations.

Gallery Sorting & WordPress Bot Enhancements:
-Added a sorting function to the wp_bot_new script.
-Each gallery is now automatically sorted by clicking the sort button, organizing images from newest to oldest after upload.

Workflow Documentation & Map Updates:
-Created documentation to explain how the full system process works, making the workflow easier to understand and replicate.
-Made changes to map settings for improved overlay accuracy and presentation.
-Completed full workflow documentation outlining every stage from image processing to upload.

Error Handling & System Robustness:
-Implemented a retry mechanism that gives the program up to 3 attempts in case of server errors.
-Worked on improving system performance to handle edge cases more gracefully.

RTL Function Update:
-Modified the RTL function to better align with the latest workflow structure and improve signal handling or recording performance.

To Do

-Monitor system logs for error frequency and assess the effectiveness of the retry mechanism.
-Continue testing the new filename format across different galleries.
-Refine and translate documentation for potential external use.
-Expand the sorting feature to support more gallery configurations if needed.