Implement "lib_extra_dirs" option for project environment

[glelouet]

My platformio.ini :

[platformio]
lib_dir = ../lib

[env:uno]
platform = atmelavr

1=onewire

lib_install = 1

I also have ../lib/mylib/lib.h that is #include in my sources
when I init, platformio downloads the onewire lib and add it to ../lib . However this leads to two issues

• ../libs and ./ are managed by git so I had to add a specific line in the .gitignore, which is weird and can lead to future mistakes
• since the ../lib is used by several projects, the .gitignore must be expressive enough to match all library used (in my case it is lib/*_ID*)
• if I want to compile with arduino IDE , I needs to change the lib_dir to ~/.arduino/libs ; but arduino IDE can't then access the ../lib libraries.

possible solution is to have two environment variables : lib_dir to specify where we PUT the required libs, and inc_dir to specify where are our private library not managed by platformio downloader.
This way, I could set inc_dir to ../lib (or ../lib, ./lib, ../../lib with a list) in my platformio.ini , and set the lib_dir in the cli if modification required (eg using arduino IDE)

[glelouet]

PlatformIO, version 2.8.4

[ivankravets]

The lib directory in project is used for the private libraries. PlatformIO Library Manager doesn't touch them. The lib_dir in platformio.ini allows to override default location for the GLOBAL libs.

I want to mention that in PlatformIO 3.0 the logic with Library Manager will be rewritten from the scratch. By default all libs will be installed in project/lib directory with semver support. If you want to install library globally, then need to specify platformio lib install -g/--global flag. In this case, the library will be installed globally and will be accessed for the all projects. See #475.

I recommend to take a look on the all lib-related issues for PlatformIO 3.0. https://github.com/platformio/platformio/issues?q=is%3Aopen+milestone%3A3.0.0+label%3Alib

The question: will PlatformIO 3.0 resolve your issue?

[glelouet]

The way you explain it, it won't, because my private library is in ../lib and not in ./lib .

In my case I need to have :

• specification of a set of PRIVATE lib directories ( eg ../lib ../../lib ) in which I manage the code (NOT managed by semver) which is not possible according to the description you did. This is my request, an inc_dir [=lib] option.
• specification of WHERE to put downloaded libs by semver, eg in ~/.Arduino/lib if I want to build for Arduino IDE.

If I understood well, the first point is not possible in pio 3 , I assume the semver lib dir can be changed so the second point would be.

[ivankravets]

How about lib_extra_dirs per [env:xxx] environment? Users will have ability to tell PlatformIO Library Dependency Finder extra directories where libraries can be located.

Currently, we have only 3 locations where LDF looks for libs https://github.com/platformio/platformio/blob/develop/platformio/builder/main.py#L91

Summary:

• [platformio][lib_dir] - will be used to change location of the lib folder per project. Currently, it changes location of GLOBAL library folder.
• there will no be ability to change location of GLOBAL LIB DIR, only using home_dir, where global lib dir == home_dir + lib.
• user can setup extra locations to look for libraries.

The order/priority:

1. Project lib directory
2. lib_extra_dirs
3. Global lib
4. Framework libs

[glelouet]

I'm not very good at python.

What I see from ( https://github.com/platformio/platformio/blob/develop/platformio/util.py )
is that

• util.get_lib_dir() returns the target download dir
• utils.get_projectlib_dir() returns the private lib dir

getprojectlib_dir should have its own resolution instead of static lib/, eg

def get_projectlib_dir():
    return _get_projconf_option_dir(
        "inc_dir",
        join(get_home_dir(), "lib")
    )

then this method could return a list of directories ( change the get_projconf_option_dirS to split the option_dir by ',' then map using expanduser) which would be appended the two other lib directories in main.py

I'm not very good at python so I don't try to do that right now, but I may give it a try if you can't afford.

[ivankravets]

Please don't look into the code. Please read my comment #537 (comment) and give me an answer about this approach in PlatformIO 3.0. I'll not change LDF in PlatformIO 2.0.

[glelouet]

Sorry I didn't get a question in previous comment, so to be constructive I looked at your link which actually was some code. It took me some time and I feel awkward when you tell me to look at code then to not to after I did look at code.

Please rephrase your question. I can rephrase my issue if you don't get my point, as it seems we don't understand each other well.

My previous comment proposed to modify code in utils.py. It was composed of two successive modifications, the second one being optionnal

• resolve utils.get_projectlib_dir() against a inc_dir variable
• make this method return a list of directories(instead of a single one), to possibly load libraries from several private directories.

Those modifications do not require to modify LDF, furthermore they only ADD functions (the inc_dir variable is optionnal) and thus do not require a change in the main version number.

[ivankravets]

@glelouet I've just tried to re-read your #537 (comment).

possible solution is to have two environment variables : lib_dir to specify where we PUT the required libs

This is equal to the default value /.platformio/lib

and inc_dir to specify where are our private library not managed by platformio downloader.

This is equal to the lib folder from the project

This way, I could set inc_dir to ../lib (or ../lib, ./lib, ../../lib with a list) in my platformio.ini , and set the lib_dir in the cli if modification required (eg using arduino IDE)

If you don't want to put any info into platformio.ini, you can make symbolic links to that folders.

[glelouet]

If you don't want to put any info into platformio.ini, you can make symbolic links to that folders.

You don't understand me, and I'm tired of trying to explain myself.
If you want to discuss, read my whole issue again. You don't get me at all, I said I DO want to change the plaformio.ini

[ivankravets]

Please provide archive with example project

[glelouet]

why ?

[ivankravets]

Dear @glelouet,

1. If you open issue and know that "PlatformIO member" You don't understand me - you have to explain in details, otherwise issue will not be resolved.
2. I've got you since the 1-st your comment. What is more, I explained you twice why it doesn't work in PlatformIO 2.0 and how could work in PlatformIO 3.0. However, you don't want listen me and answer in each comment "need to implement inc_dir for private libs".

Let me rephrase:

1. PlatformIO 2.0 has PUBLIC & PRIVATE libs support.
2. The public libs are located in PLATFORMIO_HOME/lib by default. However, user have option to change this path in platformio.ini via lib_dir.
3. The PRIVATE libs are located in the project's lib directory and you can't change it. What is more, this lib contains readme.txt with DETAILED explanation how to use it, like ...This directory is intended for the project specific (private) libraries....

---

As I've said above, I don't like current PlatformIO 2.0 scheme with these lib folders. In this case, I propose these options:

• Handle project/lib for "private" libs + for project specific libs (dependencies). Library Manager will use it to manage "local/private" libs. That is good when you have access within IDE to project's related libraries and don't need to open "external" folder with "global libs". What is more, different projects can depend on the different version of the libraries.
• You can mix "own libraries" with "library manager"'s libraries within project/ilb. If library isn't installed by library manager (your own lib), PlatformIO Library Manager will not touch it.
• platformio.ini > [platformio] > lib_dir will relate to project/lib directory, not to global as it is.
• If you need to change GLOBAL libs folder, just change platformio.ini > [platformio] > home_dir, where GLOBAL dir will equal to home_dir/lib.

*And the last... *

You want to have "OWN" PRIVATE INC dirs. No problems. See my #537 (comment). According to it you will have ability to specify OWN EXTRA LIB dirs. You can call them "PRIVATE" or even "SPONGE BOB DIRS". For example,

[env:myenv]
platform = ...
board = ...
lib_extra_dirs = /path/to/private/dir1,/path/to/private/dir2,/path/to/sponge/bob/dir

---

Now, please forget about "i want inc_dir" and try to listen what I propose. If you have corrections/comments to my proposition, I'm glad to correct it.

P.S: Guys, @FWeinb, @marvinroger, @dave-newson, @furyfire, @oskargargas, @sarfata, @jasonmhite what do you think?

[marvinroger]

You don't understand me, and I'm tired of trying to explain myself.

@glelouet I am pretty sure working on such an important _open-source_ project and getting this kind of answers is tiring too. Just saying.
Anyway, GitHub is one of these few rare places on the Web where constructive discussions are possible, so let's not bicker.

@ivankravets what you propose sounds good and reflected to me 👍

[dave-newson]

@glelouet I'm having trouble reading the original request. If I'm reading you right, you've got two concerns;

• the need to include various extra lib directories
• subsequent compatibility with the Arduino IDE.

Going back to your original requirements (I've fiddled with the names because the terminology is confusing);

• You've got shared custom code sitting in ../libs. Let's rename this to ../shared_libs for readability.
• You've got platformio libs sitting in ./libs.
• You've got private non-shared code for your project (maybe?). Let's call this ./private_libs.

My understanding is that the Arduino IDE is an ignorant beast, and it'll only include libs which sit inside ~/.arduino/libs.

lib_dir to specify where we PUT the required libs, and inc_dir to specify where are our private library not managed by platformio downloader.
This way, I could set inc_dir to ../lib (or ../lib, ./lib, ../../lib with a list) in my platformio.ini , and set the lib_dir in the cli if modification required (eg using arduino IDE)

I'm not understanding how this would solve your problem. Using this method it sounds like you would be telling PlatformIO to use/place libs from/in ~/.arduino/libs, but the inverse - being able to compile using Arduino IDE - is still not possible, as it won't resolve your paths of ../lib/whatever.

Concerning Arduino IDE compatibility, even if you tell platformio to dump libraries in the ~/arduino/libs folder, Arduino IDE still can't see your ../shared_libs because it doesn't know to look for them.

You could work around this issue by using symlinks (yay for linux) or in PlatformIO 3 you can fold ../shared_libs into its own project/repo, and use PlatformIO to semver manage that dependency.
That way your ../shared_libs becomes ./libs/shared_libs, and I'd say this is the right way (although not the quick way) to do things. Otherwise your dependency is not managed at all.

@ivankravets suggestion of using a lib_extra_dirs config will partially solve your ./libs vs ./private_libs problem. In fact you can use this solution for your ../shared_libs too, but you'll still be screwed when it comes to Arduino IDE.

I would expect the following to work just fine:

[platformio]
lib_extra_dirs = ./private_libs,../libs

Honestly, I can't see a way to fix compatibility with the Arduino IDE if you keep using the ../libs (../shared_libs) directory. You'd be safer making this its own lib, and then a direct dependency of your project. That's not currently possible in PlatformIO2, but should show up in PlatformIO3.

Bear in mind I'm not very familiar with the Arduino IDE because I find it horrible to use. If you know something we don't about how to include libraries, let us know.

[glelouet]

@dave-newson

My understanding is that the Arduino IDE is an ignorant beast, and it'll only include libs which sit inside ~/.arduino/libs.

Well, it also has a list of libraries folder to look into.

but the inverse - being able to compile using Arduino IDE - is still not possible, as it won't resolve your paths of ../lib/whatever.

following this blog ( http://www.ikravets.com/computer-life/platformio/2014/10/07/integration-of-platformio-library-manager-to-arduino-and-energia-ides ) it would. (short : setting the lib_dir to ~/Arduino/Libraries )

@ivankravets

You want to have "OWN" PRIVATE INC dirs. No problems. See my #537 (comment). According to it you will have ability to specify OWN EXTRA LIB dirs. You can call them "PRIVATE" or even "SPONGE BOB DIRS".

I didn't understand the [env:xxx] and it was about platformio 3 .
My issue is about platformio 2, I looked into the code to help you find a solution. If you say the behaviour changes, then I need to understand this new behaviour, but I don't and this is not my issue.

I only had ONE issue with platformio 2 , and this was, that I could not override the ./lib/ as being the SPONGE BOB folder. Maybe the example of code I gave you could help, maybe it was stupid ; but in the end you throw it away and this is rude.

@marvinroger

where constructive discussions are possible, so let's not bicker.

Yes but when I feel like my point is out of the discussion there is no reason for me to try to discuss. I don't want to waste time, nor to waste yours - platformio is good and I'm sure you can enhance it.

I unsuscribe, just realized I didn't get nor bring anything good from this.

[ivankravets]

@glelouet Why did you close this issue? Please read my comments above. I don't have any objections to resolve this issue but it will be done in PlatformIO 3.0. Also, please read carefully #537 (comment) and I wait for the answer: "Will PlatformIO 3.0 resolve your issue?" according #537 (comment)

P.S: You are right, let's save time each other. What is more, all PlatformIO users are equal for me. I mentioned a few PlatformIO users above "not for the help to humble @glelouet", but for the constructive answer as for #537 (comment).

I don't know these users, but they are authors/commentors of a few issues related to PlatformIO 3.0 Library Manager. See https://github.com/platformio/platformio/labels/lib

As you can see, I don't have aim to "discard all users requests" but quite the contrary I'm grateful them for the comments, ideas and corrections. The only this friendly and politely discussions in our community will help PlatformIO Team to create PlatformIO FOR THE ALL, but not only for @ivankravets.

Regards, Ivan.

[dave-newson]

@glelouet don't be discouraged, your requests are still valid. The biggest problem with this issue is the terminology and requirements are slightly confusing, so unfortunately you will need to bear with us while we confirm exactly what the issue is, and if it's something PlatformIO can/should resolve.
Sometimes it's necessary to talk slow using simple words and bullet points. Write a story, paint us a picture. If understanding people was easy we wouldn't have wars.

following this blog ( http://www.ikravets.com/computer-life/platformio/2014/10/07/integration-of-platformio-library-manager-to-arduino-and-energia-ides ) it would. (short : setting the lib_dir to ~/Arduino/Libraries )

Thanks for the link, that's very informative about what it is you're trying to accomplish.

Unfortunately, I think this still doesn't address the problem of Arduino IDE not being able to see the ../shared_libs directory, so my point still stands.
You would instead need to make your ../shared_libs into its own project, eg. glel_shared_code and then include that as a dependency in your PlatformIO project. Doing that would ensure it is added to ~/arduino/libs by PlatformIO, and then Arduino IDE would see it.

With that covered, let's ignore the issue of Arduino IDE's terrible library management and address this whole lib_extra_dirs thing.
With the libs_extra_dirs you would be able to tell PlatformIO to include libraries from a number of additional locations. This would let you have your ../shared_libs, ./libs and ./private_libs folders, while PlatformIO only installs 3rd party libs to ./libs.
(Unfortunately even with that feature Arduino IDE is still trash. You still need all your libs to live in ~/arduino/libs for Arduino IDE to see it)

I only had ONE issue with platformio 2 , and this was, that I could not override the ./lib/ as being the SPONGE BOB folder. Maybe the example of code I gave you could help, maybe it was stupid ; but in the end you throw it away and this is rude.

I think Ivan is really trying to concentrate his efforts on bringing PlatformIO3 into production, as it'll fix a massive range of issues rather than just this one concern, so he's looking for how a feature could be added to PlatformIO3 which will solve this issue, rather than continuing to work on deprecated PlatformIO2 code, which will only be thrown away in the coming months.

Unfortunately with OSS, unless you're willing to make your own contributions, it can mean waiting until the authors goals align with yours and also coming to a compromise with them.

As an aside, using CLion I achieved the same ability - adding multiple lib dirs in PlatformIO2, by adding the following configs:

platformio.ini

[env:xxx]
build_flags = -L../libs

and I think I also used:
cmakelists.txt

add_subdirectory('../libs')

Well, it also has a list of libraries folder to look into.

How do you set this? I Googled for it and I couldn't find anything.
Can you provide some details? Is it an env var or something?

///// Thanks for the clarification.

[ivankravets]

@dave-newson

build_flags = -L../libs

That is compiler/preprocessor flag. PlatformIO Library Dependency Finder doesn't handle it. We have 2 different meaning under lib in PlatformIO:

1. lib is compiler compatible library, like mylib.a, mylib.so and etc. -L is the path where compiler will look for *.a libs. What is more, you can specify which libs should be picked up via -lmylib. BUT.... This is really advanced usage and I don't recommend to use it if you don't understand what doest is mean.
2. lib is raw source code of the future library. PlatformIO makes REAL lib.a from your source code automatically. Take a look at .pioenvs/*/.

---

As for Arduino IDE & PlatformIO. @glelouet is right, we can share the same folder between PlatformIO & Arduino IDE without problems.

The issue here that @glelouet wants to keep "some private" libs between both sides. I hope that PlatformIO 3.0 will help him to achieve it.

@glelouet please answer. Thanks.

[glelouet]

I gave up arduino IDE and gave Atom+Platformio a try.
It's ok so far, still missing the features of eclipse(autocomplete is …strange) but I'm fine with it as of now.

@ivankravets

platformio.ini > [platformio] > lib_dir will relate to project/lib directory, not to global as it is.

well I wont discuss the variable name, I think separating global and project-specific libs like this is good

GLOBAL libs folder, just change platformio.ini > [platformio] > home_dir, where GLOBAL dir will equal to home_dir/lib.

Well, I think we should have a separation between platformio home_dir and the corresponding global_dir. What happens if I download one lib for different architectures ? Do they share the same lib source ?
Is there a way to expand variables in platformio.ini, like global_dir=$(home_dir)/arduino/libs ?

Besides those may-be issues, it seems good to me.

[ivankravets]

Temporary link to development version of PlatformIO 3.0 docs http://docs.platformio.org/en/feature-platformio-30/projectconf.html#lib-extra-dirs

[mohsh86]

this thread got me confused as hell, i have a couple of questions:

1. is lib_extra_dirs implemented yet in PlatformIO 2.11 ?
2. is pio lib install -g/--global is not yet implemented in PlatformIO ? i ran it on lib 89 (Pub sub) and it got installed in ~/.platformio/lib , is that the default ?
3. i have my own library that i am sharing across different project, is there any current way on current version (2) to include them into project without copying or creating symbolic link to the project lib folder ?

note: build_flags = -L

didn't work as well

[ivankravets]

@mohsh86 PIO3 pio lib install -g/--global command is equal to pio lib install command in PIO2.

You can place temporary your own libraries to ~/.platformio/lib and after PIO3 release you can create extra library storage and specify it in lib_extra_dirs.

PIO3 is planned for release tomorrow.

[mohsh86]

awesome dude, you rock !