How NOC Data Refresh Works
- The scheduler checks daily whether a refresh is needed (every 30 days)
- It scrapes all 516 NOC 2021 unit group profiles from noc.esdc.gc.ca
- Each profile is compared against the database: new profiles are inserted, changed profiles are updated
- A 500ms delay between requests ensures polite scraping behavior
- The full refresh typically takes 5-10 minutes depending on ESDC response times
- Manual refresh bypasses the 30-day cooldown but respects the 7-day minimum between runs