Allow library.json to specify sources other than PlatformIO's Repository #461

dave-newson · 2016-01-16T04:46:37Z

The Problem

If I want to use a library.json to manage my projects 3rd party dependencies, then these 3rd party libraries;

must already have a library.json file
must be registered on PlatformIOs repository.

Desired Functionality

Let's assume that, in a perfect world:

my library.json manages all the third party dependencies in my project.
after I run platformio update all dependencies are fulfilled.

The desire is to eliminate the need to manual checkouts, clones, code downloads, etc. In 90% of common use cases I want to just run platformio update and have all the magic taken care of.

Use Case 1 - I want to use a library which isn't on the PlatformIO repository

There are always going to be cases where I want to include a library in my code but the libraries author has not included a library.json file in their source.

This may be because:

they don't known about PlatformIO
they haven't gotten around to putting it in the registry
they have some objection to using PlatformIO at all

Frankly, and this is the important bit; Who cares? I should not require their consent to perform what is basically git clone. The library.json file is a helper for PlatformIOs repository, and not necessary for code to function.

The official solution to this problem is "submit a library.json PR to the 3rd party", but come on. That could take weeks to get actioned, and in the mean time I have to manually include the dep. That's hardly a solution.

Use Case 2 - I want to use a specific fork

Say I've followed the standard GitHub approach of forking, branching, creating a change and then PRing it back to the main third party project. That's nice, and I'm a good open-source citizen.

Problem is, this change won't be available in the library via PlatformIO until it gets merged back in, if it gets merged back in.

I could submit my fork up to PlatformIO's repository, so now we have "arduino-stepper-dave-newson", but frankly that's just silly. Do we want people to do that for every hotfix and branch? No, I don't believe we do.

Use Case 3: Source code isn't on VCS at all.

There's also cases where libraries have been put online as .zip files by authors born in the 50s and who still write their documentation in .txt files and put things on .edu domains. These libraries simply may never end up on a VCS like GitHub (licensing, anyone?).

The only way around this right now is a manual install.

Desired Change

This approach is stolen directly from the PHP package manager Composer.

I can specify required dependencies in PlatformIO's library.json as follows:

"dependencies":
[
    {
        "name": "derp",
    },

Please also allow me to override the libraries source with something akin to:

"repositories": [
    {
        "name": "derp",
        "url": "https://github.com/dave-newson/derp.git",
        "type": "git"
    }
],

The name property indicates to the PlatformIO executable that I want the dependency derp to come from the specified GitHub repository. This way I can use forks and branches of my own.

In additional, PlatformIO shouldn't require this repository to contain a library.json file. It should just checkout the entire repo wholesale and dump it into my project.

Again, stolen from the likes of Composer, a similar approach can be used to drag in zip archives:

"repositories": [
    {
        "type": "package",
        "package": {
            "name": "derp",
            "version": "master",
            "dist": {
                "type": "zip",
                "url": "http://example.com/src/derp.zip",
                "reference": "master"
            },
        }
    }
]

Obviously PlatformIO requires its own approach and json structure, but this would enable people to include from any git-based repository or zip archive, without the need for registration with PlatformIO.
This would greatly expand its capabilities and enable faster adoption.

This system could also allow expansion for compatibility with any other VCS like Mercurial or SVN.

The text was updated successfully, but these errors were encountered:

oskargargas · 2016-01-19T06:40:36Z

On top of that please allow checkout of specific version based on git (or other vcs) tag or commit hash.
Ruby's Bundler and iOS Cocoapods (which uses the same library file format) are perfect example whats needed in terms of dependency version management and custom sources checkout.

ivankravets · 2016-01-19T14:54:14Z

@oskargargas Of course, I'll implement it.

furyfire · 2016-01-26T11:26:03Z

The design specs for PHP's composer should apply directly to platformio.
https://getcomposer.org/doc/
Maybe it is time for the library's you require to be placed in the project folder and not platformio's user folder?

What about resolving dependecies? Can package A require package B which will also be automatically installed and updated?

ivankravets · 2016-01-26T11:34:45Z

@furyfire

Maybe it is time for the library's you require to be placed in the project folder and not platformio's user folder?

Thanks! See #475

What about resolving dependecies? Can package A require package B which will also be automatically installed and updated?

These features are partially implemented in the PlatformIO 2.0. However, this functionality will be rewritten from the scratch in PlatformIO 3.0. See https://github.com/platformio/platformio/issues?utf8=✓&q=is%3Aopen+milestone%3A3.0.0+label%3Alib

probonopd · 2016-02-08T21:58:19Z

👍

cslamar · 2016-04-23T02:43:16Z

👍

#742, #459, #542, #763, #759, #753, #757, #749, #748, #745, #519, #709, #743, #413, #498, #410, #740, #361, #414, #554, #732, #588, #475, #461, #101, #719, #721, #537, #415, #522, #289, #556, #570, #456, #617, #432, #408, #479, #667, #510)

* develop: Fix incorrect line order when converting from INO to CPP and pointer is used Fix unit test Notify about `version` field when creating library Add support for SparkFun Blynk Board Return valid exit code from ``plaformio test`` command Disable SSL Server-Name-Indication for Python < 2.7.9 Version bump to 3.0.1 (issue #772) Disable temporary SSL for PlatformIO services // Resolve #772 Version bump to 3.0.0 (issues #770, #766, #747, #730, #765, #640, #659, #742, #459, #542, #763, #759, #753, #757, #749, #748, #745, #519, #709, #743, #413, #498, #410, #740, #361, #414, #554, #732, #588, #475, #461, #101, #719, #721, #537, #415, #522, #289, #556, #570, #456, #617, #432, #408, #479, #667, #510) Fix menu height for docs Fix issue with multiple archives when linking firmware Add migration guide for PIO2 to PIO3 Search libraries by headers/includes with ``platformio lib search --header`` option Update pio run command examples Add Unit Testing Demo Update PIO Plus badge title and link Add PlatformIO Plus badge Add links to PlatformIO Plus

danielschenk · 2018-09-15T10:07:22Z

I stumbled upon this issue because I'm trying to accomplish the exact same thing as @dave-newson, but I can't manage to get it working. I'm trying to have a library.json locally in my project to download a library which is somewhere on GitHub but doesn't have a manifest.

My structure looks like this (I've heavily simplified it for the sake of clarity):

- Application.cpp <-- includes MyLocalLib1.h
- [lib]
    - [MyLocalLib1]
        - MyLocalLib1.h <-- includes something of rtmidi
    - [MyLocalLib2]
        - ...
    - [rtmidi]
        - library.json
        - rtmidi.py

library.json looks like this:

{
    "name": "rtmidi",
    "repository": {
        "type": "git",
        "url": "https://github.com/thestk/rtmidi.git",
        "branch": "v3.0.0"
    },
    "frameworks": "native",
    "build": {
        "extraScript": "rtmidi.py"
    }
}

I need the rtmidi.py because based on platform (Mac/Windows/Linux) I need to define a preprocessor symbol and some additional link flags.

I can't get it to work. When I specify rtmidi in the lib_deps of my platformio.ini, I get a warning that it doesn't exist (when running verbose).

I've considered several other solutions:

Fork rtmidi, add the 2 files, and hope they will pull it in (not ideal as the issue author pointed out)
Host my own GitHub repository with the 2 files, and point platformio lib register to there (does this even work? Will it download from rtmidi in that case?)
Clone rtmidi locally as a git submodule, in a subfolder of the folder where the manifest and script files are, but this would make the project less user-friendly (need to clone recursively) and complicate the manifest.

Am I missing something? I think it should be possible to make platformio download the library for me based on a local manifest, right?

ivankravets · 2018-09-15T11:43:09Z

@danielschenk

""frameworks": "native","

http://docs.platformio.org/en/latest/projectconf/section_env_library.html#lib-compat-mode

danielschenk · 2018-09-15T12:48:26Z

@ivankravets

When I remove that line, LDF still does not find it 😢. And I actually want to use the compatibility feature of the LDF, because I have different environments in my platformio.ini:

One which builds a little test app for my MIDI logic, which uses an USB MIDI interface and runs on the PC, therefore uses platform = native and needs rtmidi.
One which builds the application for the real target (ESP32 with Arduino framework), using the same MIDI classes but different MIDI driver (using Arduino HardwareSerial).

ivankravets · 2018-09-16T19:40:43Z

What does mean framework=native in library.json? Do you mean platform=native?

Anyway, could you provide a simple project to reproduce this issue?

danielschenk · 2018-09-16T21:14:44Z

You are right, it should have been platforms: "native", in the manifest. I tried it, unfortunately does not make a difference.

I attached a simple project to reproduce the issue. When I run it, compilation fails because including RtMidi.h fails.
platformio-issue-461.zip

ivankravets · 2018-09-17T12:58:35Z

@danielschenk how can LDF find a library if there is no #include <RtMidi.h>?

danielschenk · 2018-09-17T18:13:05Z

@ivankravets The include statement is there, in Application.cpp.

If you're talking about the actual header file RtMidi.h, you are correct, it's not there. That's because I expect that the library automatically gets downloaded from GitHub by PlatformIO (without first registering it in the database). I was under the impression that @dave-newson wanted the same, and opened this issue for it. The issue got closed, so I thought the idea had been implemented.

If it is not possible what I want, then we unfortunately misunderstood each other.

ivankravets · 2018-09-18T10:24:53Z

If it is not possible what I want, then we unfortunately misunderstood each other.

Sure, I don't understand your problem but I'm glad to help. Please create PlatformIO project which does not work but has source files. I'll try to debug it.

danielschenk · 2018-09-19T11:19:33Z

The source file is already there, in the example I uploaded. It's src/Application.cpp. And it includes RtMidi.h like said before.

I can imagine that LDF doesn't know that it needs to download RtMidi from GitHub because it doesn't know that RtMidi.h (which should be found as dependency by LDF) is actually part of that repository. But maybe a hint could be added to platform.json to make this possible? Something like:

{
    "name": "rtmidi",
    "repository": {
        "type": "git",
        "url": "https://github.com/thestk/rtmidi.git",
        "branch": "v3.0.0"
    },
    "frameworks": "native",
    "build": {
        "extraScript": "rtmidi.py"
    },
    "headers": [
        "RtMidi.h"
    ]
}

ivankravets · 2018-09-19T11:21:50Z

This already implemented. There 2 options in your case:

lib_deps = rtmidi (if library is registered in PlatformIO registry)
lib_deps = https://github.com/thestk/rtmidi.git#v3.0.0

danielschenk · 2018-09-19T11:53:18Z

Thanks, I could not find option 2 in the documentation. I will try it out later today :)

Will it still make use of my local library.json? Because I need to run the extra script for the RtMidi build flags.

ivankravets · 2018-09-19T13:04:41Z

Yes, it will use library.json it the last one included in a library.

See http://docs.platformio.org/en/latest/projectconf/section_env_library.html#projectconf-lib-deps

danielschenk · 2018-09-21T18:52:40Z

That's still not exactly what I want. I want to use the original RtMidi from GitHub, but I want library.json and the extra script to be in my own project (because I see no point in forking RtMidi just for that).

I attached a new example to make this clear. When I run it, it tries to build a library directly (with no files) and therefore stops with an error. Instead, I expect it to:

Understand that it needs to clone RtMidi from GitHub first
Then build it with the flags as specified by rtmidi.py.

platformio-issue-461-example2.zip

ivankravets · 2018-09-23T19:47:54Z

LDF depends on library.json manifest. We don't do 3rd party work. If manifest exists, we use this library and don't check source files. There a lot of ways how to resolve your issue:

Fork rtmidi librart and update with own library.json and extra script. Later, use direct GIT link to your repo
Add "lib_deps = https://github.com/thestk/rtmidi.git", PIO will clone this repo, then use PRE:SCRIPT and put into the library 2 files: library.json and your extra scripts per library

ivankravets added this to the 3.0.0 milestone Jan 16, 2016

ivankravets added feature ldf Library Dependency Finder labels Jan 16, 2016

ivankravets self-assigned this Jan 16, 2016

ivankravets mentioned this issue Jan 31, 2016

PlatformIO 3.0 Roadmap #423

Closed

37 tasks

ivankravets mentioned this issue Feb 8, 2016

How to specify library dependencies in platformio.ini #497

Closed

ivankravets closed this as completed in 758396c Aug 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow library.json to specify sources other than PlatformIO's Repository #461

Allow library.json to specify sources other than PlatformIO's Repository #461

dave-newson commented Jan 16, 2016

oskargargas commented Jan 19, 2016

ivankravets commented Jan 19, 2016

furyfire commented Jan 26, 2016

ivankravets commented Jan 26, 2016

probonopd commented Feb 8, 2016

cslamar commented Apr 23, 2016

danielschenk commented Sep 15, 2018

ivankravets commented Sep 15, 2018

danielschenk commented Sep 15, 2018

ivankravets commented Sep 16, 2018

danielschenk commented Sep 16, 2018

ivankravets commented Sep 17, 2018

danielschenk commented Sep 17, 2018

ivankravets commented Sep 18, 2018

danielschenk commented Sep 19, 2018

ivankravets commented Sep 19, 2018

danielschenk commented Sep 19, 2018

ivankravets commented Sep 19, 2018

danielschenk commented Sep 21, 2018

ivankravets commented Sep 23, 2018

Allow library.json to specify sources other than PlatformIO's Repository #461

Allow library.json to specify sources other than PlatformIO's Repository #461

Comments

dave-newson commented Jan 16, 2016

The Problem

Desired Functionality

Use Case 1 - I want to use a library which isn't on the PlatformIO repository

Use Case 2 - I want to use a specific fork

Use Case 3: Source code isn't on VCS at all.

Desired Change

oskargargas commented Jan 19, 2016

ivankravets commented Jan 19, 2016

furyfire commented Jan 26, 2016

ivankravets commented Jan 26, 2016

probonopd commented Feb 8, 2016

cslamar commented Apr 23, 2016

danielschenk commented Sep 15, 2018

ivankravets commented Sep 15, 2018

danielschenk commented Sep 15, 2018

ivankravets commented Sep 16, 2018

danielschenk commented Sep 16, 2018

ivankravets commented Sep 17, 2018

danielschenk commented Sep 17, 2018

ivankravets commented Sep 18, 2018

danielschenk commented Sep 19, 2018

ivankravets commented Sep 19, 2018

danielschenk commented Sep 19, 2018

ivankravets commented Sep 19, 2018

danielschenk commented Sep 21, 2018

ivankravets commented Sep 23, 2018