2. Data Analysis Methodologies and Procedures
2-7. Handling Missing Items
1) Current Reality and Actual Conditions of Customer Data
- A. Bloating of Product Masters:
Product masters contain discontinued and historical items, meaning registered item counts are typically 2 to 3 times greater than items with actual shipping records. To improve analytical accuracy and processing speed, narrow down and extract items that had actual shipping records during the target period as much as possible.
- B. Deficiencies in Product Dimensions and Shape Data:
In field operations where volume and weight management are not currently performed, dimension data in master records are often unmaintained (or blank/inaccurate), so they cannot be swallowed as-is.
- C. Discrepancy in Destinations:
"Ordering Clients" on data do not necessarily match "Actual Delivery Destinations (Shipping Locations)," so aggregation by shipping location unit aligned with field operations is necessary.
- D. Proprietary Expressions & Special Transactions (e.g., Issuance Being Receipt):
Industry- or company-specific customary expressions exist, such as registering as "Issuance" when the product management department purchases in bulk and distributes to various centers (effectively an "Inbound Receipt" to the center). Perform data cleansing after properly understanding the background and meanings of these items.
2) Patterns of Missing Data and Irregular Data
- A. Zero-Shipping & Negative-Shipping Data:
Cases where records with zero or negative quantities are included due to inter-client product transfers, post-instruction cancellation processing, or inclusion of return data.
- B. Item Exists but Data Unentered (Null):
Cases where case pack quantities, volume, or weight data are left blank.
- C. Partial Data Omission / Unentered Values:
Cases where only specific dates, destination codes, or item categories are missing.
3) Concrete Countermeasures for Missing Data
- Establishing Rules Through Meetings with Customers:
Decide handling rules for unentered/inconsistent data (e.g., average value imputation, exclusion conditions for specific records) through prior consultation with customers and operational managers.
- Salvaging and Utilizing Data:
Even if partial omissions exist, process and supplement data estimable from other items (salvageable data) as much as possible for use in analysis.
- Consistency Verification of Total Data Aggregations:
Compare and verify aggregated totals after applying exclusion rules against aggregated values of the original overall dataset (control totals), ensuring the reasons for discrepancies can be clearly explained.
- Executing Analysis After Cleansing:
Execute accurate analytical processing against the "Valid Analysis Target Data" organized based on established rules.