Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce GLS Pilot blogs. #295

Merged
merged 5 commits into from
Jan 4, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added blog/2025-01-09-antonius/antonius_app.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added blog/2025-01-09-antonius/antonius_excalidraw.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added blog/2025-01-09-antonius/antonius_h3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
61 changes: 61 additions & 0 deletions blog/2025-01-09-antonius/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
slug: streamlining-the-design-of-parcel-delivery-routes-with-h3
title: "Streamlining the design of parcel delivery routes with H3"
authors: [antonius]
tags: [introduction]
unlisted: true
image: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/antonius_social.png
hide_table_of_contents: false
keywords: [introduction]
---

In the parcel delivery business, geospatial analyses are crucial to answer questions about daily operations. Do delivery drivers visit the same regions each day, letting them know their areas intimately? Or is there a high volatility of the regions? And of course, how do we optimize the routes of multiple drivers servicing the same region?

Those are the questions Antonius is working on at GLS Studio, an innovation lab by GLS ([General Logistics Systems](https://www.gls-us.com/)), an international parcel delivery service provider.

While the planned areas of driver tours can be rather well-defined, an evaluation of the areas actually served by drivers is equally important and might not be as easy. It can be used for guiding delivery drivers to become more efficient while also enabling managers to identify planned areas that are suboptimal due to built environment features like rivers or big highways that have not been considered in the planning.


In this blog post, Antonius highlights how he uses Fused to create stable delivery areas for single-day and multi-day aggregates.

import ReactPlayer from 'react-player'

<ReactPlayer playsinline={true} className="video__player" playing={true} muted={true} controls height="100%" url="https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/driver_areas_across_resolutions.mp4" width="100%" />



{/* truncate */}


## The Challenge Designing Delivery Areas

Creating delivery areas out of single geospatial data points is challenging. A naïve approach would be to use convex polygons. This causes multiple issues: a single data point can have a high influence on the shape and area of the polygon, a tour consisting of multiple not connected sub-areas is hard to detect and correctly display, and just as important this limits the area to what is covered by historic data, so when intending to use areas for the future, there remain gaps which have not been covered historically. Lastly, calculations using polygons are computationally expensive, hindering ad-hoc changes to the selected time span or the calculation parameters.


A further challenge is that building new and sometimes experimental features means that the database is often not optimized for the use case. Data needs to be joined between multiple tables and even between multiple data sources. Therefore, fetching all data can take a considerable amount of time, highly limiting the usefulness of having an experimental feature to play around with.



## Solving the Challenge with H3 and Fused UDFs

The solution to the first problem came by using H3 cells instead of polygons. By assigning cells to the drivers based on their historic deliveries to the cell, driver areas result automatically. Using H3 cells across different resolutions also allows us to represent the differences between urban and rural areas. While there exists one ‘base resolution’ to ensure non-overlapping and complete areas, the logical hierarchy among H3 cells can be used to calculate on lower resolutions for rural areas that see fewer deliveries, speeding up the computation as well as ensuring a broader coverage of those areas beyond the historical data points. On the other hand, disputed H3 cells can be broken down to a higher resolution and assigned to different drivers or, when the ‘base resolution’ has been reached, assigned to the driver delivering most parcels to the cell. As H3 cells have clearly defined neighborhoods, areas can easily be extended beyond their historical limits when desired, covering the empty space around to include a new parcel that falls outside of historically served area boundaries.

import Image1 from '/blog/2025-01-09-antonius/antonius_excalidraw.png';

<div style={{textAlign: 'center'}}>
<img src={Image1} alt="File" style={{}} />
</div>

Fused UDFs helped us solve problems around the latency of querying and calculations. When a user looks at an area for a day, they are probably interested in the same area on some of the previous and following days as well, right? So why not pre-calculate that already? Using Fused, it is simply a matter of fire-and-forget to trigger the UDF with parameters for some previous and following days which are then already running to cache. So when the user views an adjacent day, the data will already be there. And more in general, when it is possible to limit the number of parameter combinations in an experimental feature to a manageable amount, this fire-and-have-it-cached approach is not limited to caching data from previous and following days, but can also be used for a range of other cases.



import Image2 from '/blog/2025-01-09-antonius/antonius_app.png';

<div style={{textAlign: 'center'}}>
<img src={Image2} alt="File" style={{}} />
</div>

## Conclusion

When developing new features that are not yet supported by the current data infrastructure, Fused UDFs enabled us to easily test things without having to change the underlying infrastructure in advance. The UDFs are easily shareable and adjustable, allowing testing by multiple people without having to run code locally while automatically making sure everyone is using the same code that is hosted in the UDF. And because we can easily call UDFs with HTTP endpoints, when we have verified the feasibility of a feature, it’s easy to integrate into our product.
107 changes: 107 additions & 0 deletions blog/2025-01-14-kyle/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
---
slug: how-pilot-fiber-creates-internal-tools-to-support-telecom-operations
title: "How Pilot Fiber creates internal tools to support telecom operations"
authors: [kyle,nelina]
tags: [introduction]
unlisted: true
image: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/pittman_social.png
hide_table_of_contents: false
keywords: [introduction]
---

[Pilot Fiber](https://www.pilotfiber.com/) is a commercial [Internet Service Provider](https://en.wikipedia.org/wiki/Internet_service_provider) primarily serving customers in New York City. Our primary value proposition in competing with national-scale ISPs is our commitment to customer service– we prioritize when incidents interrupt your service and will immediately jump into action to address the cause.



{/* truncate */}

### The Problem



Almost all fiber optic cables in Manhattan run through a shared manhole-and-duct system. As such, road construction or work by other providers in a manhole has the potential to damage the equipment of multiple providers. Because of this, Pilot uses an active fiber monitoring system across our network: devices located in our data centers are constantly shooting light down the fibers in our network and looking for any anomalies by comparing with a reference a “snapshot” created when that fiber was initially installed to a building.


When an anomaly is registered, it immediately fires an alert giving a fiber route and distance to the potential problem (i.e. “There is a light loss on the fiber running to 1234 5th Ave at a distance of 2.351 kilometers from the data center.”). When this happens, our engineering and support teams analyze the data within minutes to determine the exact location of the issue and, if necessary, get crews headed to the site to begin repairs.


## The Process

Historically, this process would require the attention of an Outside Plant engineer with access to specialized software and network knowledge, regardless of the time of day or day of the week. This process had known friction and a single point of failure. Today we use Fused to make this information accessible to more support team members and automate the creation of the reports field crews need to address the issue when they arrive on site.


Using Fused as a back-end glue layer, we built a web app that allows a less technical user to select a route and distance and map where the system says the fault occurred, view nearby network infrastructure, and automatically generate the necessary field reports based on that nearby infrastructure.


The workflow requires a series of calls to UDFs that act as intermediaries to a Postgres/PostGIS database, which in turn is sourcing data from other internal sources. This structure allows us to easily keep the business logic organized at the UDF layer while limiting the scope of data access and security via Postgres and internal processes maintaining the sync.


The basic process is seen below:

import Image1 from '/blog/2025-01-14-kyle/kyle1.png';

<div style={{textAlign: 'center'}}>
<img src={Image1} alt="File" style={{}} />
</div>

_Workflow diagram._


When the user selects a route and a distance to process, two separate processes are initiated. One retrieves the selected route to load onto the Mapbox-based map within the app, while the second kicks off a processing chain. This chain is utilizing UDFs that both assist in isolating the location of the fault and relevant nearby infrastructure as well as adding elements to the map display to assist the user in visualizing what may be occurring.

UDF - returning the route itself that will show up on the map.
UDF get fault point php
UDF - putting the red dot on the map.
UDF - generate the splice report.

When the user selects a route and a distance to process, two separate processes are initiated. One retrieves the selected route to load onto the Mapbox-based map within the app, while the second kicks off a processing chain. This chain calls UDFs that assist in isolating the location of the fault and relevant nearby infrastructure as well as adding elements to the map display to assist the user in visualizing what may be occurring.


import Image2 from '/blog/2025-01-14-kyle/kyle2.png';

<div style={{textAlign: 'center'}}>
<img src={Image2} alt="File" style={{}} />
</div>
_UDF to find fault location._


After calculating the likely fault location, the tool loads splice cases and fiber slack loops along that route that point to where problems are most likely to have occurred and capacity enabling faster repair of more significant damage, we next determine which splice cases are closest to the likely damage point and if any of those are within 150m of the automated distance calculation.


Once we have determined the relevant splice cases, we use Fused to build a CSV that reproduces what a splice report export looks like from our primary fiber mapping software and are able to submit that CSV to further processing to generate the Excel file that ultimately goes to the field teams detailing for them what to expect when working with the case– what cables enter the case, what fibers on which cables are spliced to which fibers on other cables, and what building and/or customer circuits are being carried along which fibers. This gives them the confidence to quickly address issues while minimizing the risk of damaging other circuits, as well as giving them the information needed to monitor services in the case while they work.

import Image3 from '/blog/2025-01-14-kyle/kyle3.png';

<div style={{textAlign: 'center'}}>
<img src={Image3} alt="File" style={{}} />
</div>
_UDF to create splice report._


## The Impact Fused

The impact of Fused across this process is many-fold:
- The ability to easily work with data across several systems.
- Centralizing business logic to the UDFs involved, eliminating the tendency for this logic to be spread across client-side processes, server-side processes, and possibly the database itself.
- Modularization of operations into UDFs. For example, the UDF that generates the splice reports for this process can easily be reused in any other process that requires the same functionality.
- The easy ability for the same UDF to simultaneously serve as a modular processing unit where needed in one workflow and a map service for display in another.
- The ability to use Python and standard libraries to enable developers who may not be spatial data experts to read, understand, and modify UDFs as necessary.

To illustrate the composition of UDFs, this image shows a map generated by 4 UDFs:
- UDF to returnin the route itself that will show up on the map
- UDF to get fault point php
- UDF to place the red dot on the map
- UDF to generate the splice report

import Image4 from '/blog/2025-01-14-kyle/kyle4.png';

<div style={{textAlign: 'center'}}>
<img src={Image4} alt="File" style={{}} />
</div>
_Composed map view._

## Conclusion

Providing non-technical users a way to have access to the data in Postgres enables anybody at the company to gather the information to get the splice team headed in the right direction as fast as possible with the latest information on what they're looking for and what they're looking at when they get there. All this person would need to input is the distance along the fiber and the affected building. By using these different UDFs and Postgress the tool will figure out the point and if relevant, it will generate the documentation that they need to give to the field crew. This improves the turnaround time of dispatching crew to the site.

Traditionally, I could do a lot of this just with Python but there were no visual maps to work with in these scenarios. Fused works so easily with the map viewer and the calculations also give a lot of context to the person running it. So even our more technical personnel can visually explore the data to immediately gain situational awareness.
Binary file added blog/2025-01-14-kyle/kyle1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added blog/2025-01-14-kyle/kyle2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added blog/2025-01-14-kyle/kyle3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added blog/2025-01-14-kyle/kyle4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 23 additions & 0 deletions blog/authors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -159,3 +159,26 @@ jacob:
title: Senior ML Engineer @ VIDA
url: https://www.linkedin.com/in/jacobbieker/
image_url: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/authors/jacob.jpeg

kyle:
name: Kyle Pittman
title: Sr. Director, Systems and Network Strategy @ Pilot Fiber
url: https://www.linkedin.com/in/kwpittman/
image_url: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/authors/kyle.jpeg

nelina:
name: Nelina Huang
title: Data Associate II @ Pilot Fiber
url: https://www.linkedin.com/in/nelinahuang/
image_url: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/authors/nelina.jpeg

antonius:
name: Antonius Moosdorf
title: Data Scientist @ GLS
image_url: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/authors/antonius.jpeg

maribel:
name: Maribel Hernandez
title: Professor @ CINVESTAV
url: https://www.linkedin.com/in/maribel-hern%C3%A1ndez-758b24134/
image_url: https://fused-magic.s3.us-west-2.amazonaws.com/blog-assets/authors/maribel.jpeg
Loading