satshkd-vercel/docs/missing_dates.md

import pandas as pd
import json

with open("historical") as f:
    json_data = json.load(f)

data_dict = {d["date"]: d for d in json_data}

df = pd.DataFrame(data_dict.values())
df["date"] = pd.to_datetime(df["date"], format="%Y-%m-%d")
df.set_index("date", inplace=True)

min_date = df.index.min()
max_date = df.index.max()
all_dates = pd.date_range(start=min_date, end=max_date, freq="D")
all_dates_df = pd.DataFrame(index=all_dates)

merged_df = pd.merge(all_dates_df, df, how="left", left_index=True, right_index=True)

missing_dates = merged_df[merged_df.isnull().any(axis=1)].index.strftime("%Y-%m-%d").tolist()
print(f"Missing dates: {missing_dates}")

Importing necessary libraries

import pandas as pd # Pandas for data manipulation import json # JSON for reading JSON file

Read the JSON data from a local file

with open("historical") as f: json_data = json.load(f) # Reading data from file

Convert the list of dictionaries to a dictionary

data_dict = {d["date"]: d for d in json_data} # Creating a dictionary with date as key and dictionary as value

Load the dictionary into a Pandas DataFrame

df = pd.DataFrame(data_dict.values()) # Converting dictionary values to DataFrame df["date"] = pd.to_datetime(df["date"], format="%Y-%m-%d") # Converting date column to datetime format df.set_index("date", inplace=True) # Setting index to date column

Create a new DataFrame with all dates between the earliest and latest dates in the original DataFrame

min_date = df.index.min() # Getting minimum date from index max_date = df.index.max() # Getting maximum date from index all_dates = pd.date_range(start=min_date, end=max_date, freq="D") # Creating range of all dates all_dates_df = pd.DataFrame(index=all_dates) # Creating DataFrame with all dates as index

Merge the original DataFrame with the new DataFrame containing all dates to find any missing dates

merged_df = pd.merge(all_dates_df, df, how="left", left_index=True, right_index=True) # Merging DataFrames to get missing dates

Print any missing dates in the dataset

missing_dates = merged_df[merged_df.isnull().any(axis=1)].index.strftime("%Y-%m-%d").tolist() # Filtering missing dates from merged DataFrame print(f"Missing dates: {missing_dates}") # Printing list of missing dates

The code imports necessary libraries - Pandas and JSON, reads JSON data from a local file, creates a dictionary with date as key and dictionary as value, converts the dictionary to a Pandas DataFrame, sets the index of the DataFrame to the date column, creates a new DataFrame with all dates between the earliest and latest dates in the original DataFrame, merges the original DataFrame with the new DataFrame containing all dates to find any missing dates, filters missing dates from the merged DataFrame, and finally prints a list of the missing dates.