git synchroniztion #24
Comments
|
Google Drive synchronization is wanted too. |
|
You can open a separate issue for Google Drive sync, thanks! |
|
@IvanMalison When you say git synchronization, were you just talking about committing to a local repo, or also potentially pulling/pushing as part of the sync from that repo to a remote? I'm just wondering what the full scope of this feature would be. I like the idea of having the "git client" handle the conflicts during a replay as part of pull-before-push, perhaps adding a new commit with the conflict left in it verbatim, so that I can resolve it later. Like: I think this would be a huge improvement from the limited options of force-load/force-save that Orgzly currently provides. |
|
Shouldn't this be done by a separate program? Or for example make a library from the core features of: https://github.com/maks/MGit and then use those in this project to provide nice integration. |
|
I use Resilio for phone-to-computer sync. Orgzly-to-sdcard is the kind of auto sync I want. Git might be a good backend for conflict resolution, but staying in sync automatically will outright prevent many conflicts from ever happening.
El 3 de mayo de 2017 7:15:32 GMT+01:00, Jappie Klooster <notifications@github.com> escribió:
…Shouldn't this be done by a separate program? Or for example make a
library from the core features of: https://github.com/maks/MGit and
then use those in this project to provide nice integration.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
|
@mkaito That would prevent the user to edit a file that is not up to date ie. no offline mode. The thing is that different users want different ways of synchronizing Orgzly with their Emacs. Git is one backend of interest but I think it's more of something like Tasker's job to update the repository. Automatic synchronizing like you suggest makes a lot of sense with Google Drive on the other hand (like Google Drive documents), but then again you would need to handle the cases where the user edit a document while offline. |
|
I think the constant need to manually synchronize are a bigger concern than the occasional lack of connection.
I'm not saying there won't ever be conflicts, just that most happen when you forget to hit the sync button.
El 3 de mayo de 2017 16:04:03 GMT+01:00, Mathieu Marques <notifications@github.com> escribió:
…
@mkaito That would prevent the user to edit a file that is not up to
date ie. no offline mode.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
|
@mkaito Well then pick another sync backend than Git :-) |
|
I just said that using git would make merges a lot more painless when conflicts do happen.
El 3 de mayo de 2017 16:33:56 GMT+01:00, Mathieu Marques <notifications@github.com> escribió:
…
@mkaito Well then pick another sync backend than Git :-)
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
|
So I set this up with MGit and then use the file system to synchronize, this works ok-ish although the workflow is a little cumbersome because you have to switch apps to commit and push and pull. So what is expected of this issue precisely?
Personally I'd prefer an auto synchronize thing, and let the advanced stuff be done with an app like MGit. You thoughts please. |
|
This issue is about automating what the "synchronize" button does. Nothing else. Git came up as a backend system to use for conflict resolution, not as an additional app that you manually use.
El 9 de mayo de 2017 16:30:20 GMT+01:00, Jappie Klooster <notifications@github.com> escribió:
…So I set this up with MGit and then use the file system to synchronize,
this works ok-ish although the workflow is a little cumbersome because
you have to switch apps to commit and push and pull.
What currently happens is I press synchronize and orgzly writes out to
the MGit repo (I assume that before that the changes exist inside the
app assigned memory). Then I do git stuff with MGit.
So what is expected of this issue precisely?
+ Do you want to auto detect we have a git repo and when you press
synchronize automatically commit push and pull just like the github app
on windows?
+ Do you want to make a MGit like UI inside orgzly?
+ Something else?
Personally I'd prefer an auto synchronize thing, and let the advanced
stuff be done with an app like MGit.
You thoughts please.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
|
So when the user presses the synchronize button, git will commit local changes, pull in remote changes and push local changes. If everything goes according to plan, otherwise panic and let the user figure it out? |
|
We could figure out what specifics work best. Perhaps stash, pull, stash pop, commit and push? Or commit, fetch, rebase?
El 9 de mayo de 2017 17:20:42 GMT+01:00, Jappie Klooster <notifications@github.com> escribió:
…So when the user presses the synchronize button, git will commit local
changes, pull in remote changes and push local changes. If everything
goes according to plan, otherwise panic and let the user figure it out?
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
|
I wouldn't want to do the stash approach, because I think it will prevent conflict detection, and I do believe on a conflict the user should be consulted, rather than choosing for the user. Then I think automatic merges are better than rebases, because a merge is non-destructive. At least according to the tutorial I just read on rebases. |
yes, definitely. automatic rebases are a bad idea |
Merges are non-destructive in the sense that they don't alter commit history, but if a merge doesn't apply cleanly, the merge commit is potentially destructive if the user doesn't resolve the conflicts correctly. A rebase is destructive in the sense that it alters the commit history of the branch that's rebased, but a rebase that applies cleanly is non-destructive to the data within each commit. |
Nope, its never destructive, because EVERYTHING that was there before is still there (i.e. the commits exist in their original context). If the user doesn't resolve conflicts correctly, its easy to go back to the states before the commit and perform the merge again.
Hmmm. define cleanly. If you take the above scenario I mentioned (i.e. a conflict resolution takes place, but it ends up being incorrect, it is actually impossible to recover the original commit in its original context). The conflict resolution isn't even recorded as conflict resolution (as it is with merges), but as part of the existing commits. If by cleanly you mean "without dropping into conflict resolution", even that is not really true. Git IS NOT a patch oriented VC, which means the "contents" of a commit is actually the entire state of the repository at that time, not something like a patch file that you might apply to a repository. Git can sort of make it seem liek this is what it stores when you look at diffs, but this is not the case. |
Yes, in the context of the entire commit history, it's non-destructive. But in the context of the merge commit itself, it is destructive. And especially in the context of Org files, it's easy to delete data and never notice it's missing. Resolving conflicts on a smartphone screen is likely to be error-prone. Especially consider cases where the user may be in a low-visibility or unstable environment, like in a moving vehicle, in direct sunlight, etc: it's not difficult to imagine a scenario where a user accidentally taps the wrong hunk, resolves the conflict, and never notices that the wrong one was selected. In such a case, having the commit history isn't going to save him--at least, it's not guaranteed to, because first he has to notice the data is missing. This has actually happened to me a few times: data seemed to be missing from an Org file, and I was thankfully able to dig it out of a git commit from months earlier. But who knows if that's happened without my noticing it. Even so, finding the needle commit in a haystack of thousands of commits over months or years is not an easy task, even with The bottom line for me is that doing git merges on a smartphone or tablet is probably a bad idea in general; and it's an especially bad idea for anyone not already familiar with git merges. If someone wants to implement such a feature for "advanced users," cool, but I think it would be a mistake to promote it to "normal" users.
A clean rebase is one that git processes automatically, without conflicts. Or, in git terms, a fast-forward.
Git is not darcs in that it fundamentally does not operate on patch theory, but it certainly is patch-oriented for some operations. Git was designed to work with patches from the beginning, as it was designed for the kernel and the LKML. It simply generates patches dynamically when needed, or uses them when they're given. |
So the way I interpreted non-destructive is 'loss of information'. Sure you can fuckup the merge commit badly, but you didn't loose any information (can always go back with a checkout and try again).
My idea was to just not even support this. If a conflict occurs, we panick and roll back. We only accept fast forwards and automerges.
I guess you mean the manual stuff, with which I agree, you shouldn't do that on the phone. Alternatively on a panick you could offer to push local changes to the remote on a branch. This allows user to merge manually on a better suited device. |
Sure, that doesn't change my point, which is that when it comes down to it, EVEN cleanly applied rebases ARE destructive, and because of the way org works, I can imagine scenarios where a cleanly applied rebase could have undesireable results (think subheadings etc.), which would make it important to have the exact original context for a change.
I don't think anyone was advocating this sort of thing. @jappeace and I were simply suggesting that going with merges here would be better than going with rebases. |
|
We find ourselves in violent agreement. :) |
|
Great discussion all! I have one minor correction, and a suggestion. Firstly the correction:
Sorry, but that's not quite right. In the git world, fast-forwarding implies that new commits are not being created, but it is possible, and in fact more common, that rebases will happen cleanly without conflicts and yet still create new commits. Actually it doesn't really make much sense to talk about fast-forwards in the context of rebasing; it's usually more applicable to merges and cherry-picks. Secondly, the suggestion: I agree that automatic syncing would drastically reduce the frequency of conflicts. But automatic syncing is not mutually incompatible with using git - you can have your cake and eat it! Have any of you guys looked at the |
I don't think git annex is a good solution since it doesnt keep the actual content of the files in the git repository. Since we ARE dealing with text files and not files that are too large to handle a simple automatically sychronized repository shoudl be good enough (basically https://github.com/hbons/SparkleShare) |
|
I'm going to take a swing at this. One issue that I forsee from looking at the existing sync code is that its not going to be super easy to batch up changes to multiple files in a single commit because there is no "finalize synchronization action". I'm assunming this is probably fine. |
Nice!
Yeah, though doing a single push would be nice. |
|
@nevenz I'm thinking about maybe making a change to the repo api to allow synchronization to be done more safely with git. I want to ensure that synchronization is only ever performed on top of the appropriate revision of a previous version of the file, so I would like the storeBook method to also take as an argument the previous revision. Does this sound resonable? |
|
@nevenz Another issue I'm having is that I would like to allow the user to specify a location on disk in which to store the git repo IN ADDITION to the actual uri that will be used to indentify the repo (which will also be used as the uri that is stored on the book). This doesn't fit in well with the existing system of having exactly 1 uri for each book. |
I'm seeing different issues every time I try it. The last time, for example, it said it uploaded a file while silently failing due to missing push permissions. I have to go through it all before I make it widely available and add a bunch of tests for various scenarios. But I'll add a preference in Settings buried under App somewhere to enable it for the next release. |
Added for #24 under Settings / App / Developer options.
|
Today I started getting Is it possible to coerce jsch to use rsa-sha2-256/rsa-sha2-512 instead of sha1? I managed to fix it by creating an ECDSA key in PEM format. PEM is still required and ed25519 still doesn't work (I thought it would from the jgit bump?) |
|
@dwoffinden Please see #904. (Apparently, I was wrong to assume that Jsch only supports RSA keys. But we should be moving to the Apache library, regardless.) |
|
I tried to use this new feature but I cannot advance from this menu. The git option does not open or do anything. Edit: I'm using the play store version in a unrooted phone.
|
|
@maikol-solis I'd check the logcat, but for me I needed something like #910 for it to work |
Sorry for my ignorance @dwoffinden ,but how do I get the logcat for orgzly? |
|
I'm getting the same as @maikol-solis, on I go to "Sync" → "Repositories" → "+" → and click on the git option, where it asks me for permission to access my files, I permit it, and then I get nothing. To see the logs, I installed CatLog from F-Droid and gave it root permission to read logs. When I filtered for "orgzly" at the time of error, it gave: |
|
I am getting the following msg saying "unkown error while cloning" what should I do? |
|
Why don't I have git as a sync option under repositories when other people seem to have had it for a while? I've tried the Play Store version and the apk files (latest and latest beta). |
I imagine those who have git as an option for sync either compiled the app from source located in this repo, or it has to do with what version of android your phone is currently running. |
Orgzly > Settings > App > Developer options > Git repository type |
AND boom goes the dynamite. |
Oh wow, I had no idea! And I’ve been missing this feature for a long time … However, I just tried it and it didn’t work. Nothing happened after I entered all of the repo information and clicked |
|
@wuqui I am not able to use it with the ssh also soo ur not missing out much |
|
I haven't tested the git sync yet because I haven't been able to set correctly the ssh repo. Maybe a little guide in the README about, it would help to test further this awesome feature. Best. |
|
Building on my previous attempt (#24 (comment)), I tried again (v1.8.10). Setup Before Attempt
Attempt
It seems to say it can't connect, but I tried this both on Wifi and Cell. Still, this is further than I got last time. |
|
@amberin That's beautiful! I looked at your build actions to see if I could find the artefact apk to download, and checked your repo for any releases but can't find anything. Is there an easy way for you to officially share your build apk? |
|
Not that I'm aware of, sorry. I just used Android Studio to install it on my phone. Even if I can pull out the apk, I wouldn't know where to upload it.
…On December 2, 2022 3:49:59 PM GMT+01:00, mtekman ***@***.***> wrote:
@amberin That's beautiful! I looked at your build actions to see if I could find the artefact apk to download, and checked your repo for any releases but can't find anything. Is there an easy way for you to officially share your build apk?
--
Reply to this email directly or view it on GitHub:
#24 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
I am also interested on this apk. Maybe you can upload it here to test it https://www.apkmirror.com/apk-upload/ Best. |
You should find the artifacts in the PR of the orgzly repo: https://github.com/orgzly/orgzly-android/actions/runs/3610790022 |
|
@tillmann Did you see this comment? #963 (comment) |
|
ehm, #287 |
Yeah... I just noticed. Truth be told, you got further than I did. My version of orgzly, only shows the button, and the button doesn't do anything. I was still using dropbox to sync my files. I don't know the specifics, but there is a community driven fork of this project. https://github.com/orgzly-revived/orgzly-android-revived perhaps something can be done to correct this there. |
…ards
You have three options:
- Suspend: It stops it from showing up permanently until you reactivate it through the browser.
- Bury: Just delays it until the next day.
- Delete: It deletes it forever.
Unless you're certain that you are not longer going to need it, suspend it.
feat(anki#Configure self hosted synchronization): Configure self hosted synchronization
Explain how to install `anki-sync-server` and how to configure Ankidroid
and Anki. In the end I dropped this path and used Ankidroid alone with
syncthing as I didn't need to interact with the decks from the computer. Also the ecosystem of synchronization in Anki at 2023-11-10 is confusing as there are many servers available, not all are compatible with the clients and Anki itself has released it's own so some of the community ones will eventually die.
feat(bash_snippets#Loop through a list of files found by find): Loop through a list of files found by find
For simple loops use the `find -exec` syntax:
```bash
find . -name '*.txt' -exec process {} \;
```
For more complex loops use a `while read` construct:
```bash
find . -name "*.txt" -print0 | while read -r -d $'\0' file
do
…code using "$file"
done
```
The loop will execute while the `find` command is executing. Plus, this command will work even if a file name is returned with whitespace in it. And, you won't overflow your command line buffer.
The `-print0` will use the NULL as a file separator instead of a newline and the `-d $'\0'` will use NULL as the separator while reading.
How not to do it:
If you try to run the next snippet:
```bash
for file in $(find . -name "*.txt")
do
…code using "$file"
done
```
You'll get the next [`shellcheck`](shellcheck.md) warning:
```
SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
```
You should not do this because:
Three reasons:
- For the for loop to even start, the find must run to completion.
- If a file name has any whitespace (including space, tab or newline) in it, it will be treated as two separate names.
- Although now unlikely, you can overrun your command line buffer. Imagine if your command line buffer holds 32KB, and your for loop returns 40KB of text. That last 8KB will be dropped right off your for loop and you'll never know it.
feat(pytest#Stop pytest right at the start if condition not met): Stop pytest right at the start if condition not met
Use the `pytest_configure` [initialization hook](https://docs.pytest.org/en/4.6.x/reference.html#initialization-hooks).
In your global `conftest.py`:
```python
import requests
import pytest
def pytest_configure(config):
try:
requests.get(f'http://localhost:9200')
except requests.exceptions.ConnectionError:
msg = 'FATAL. Connection refused: ES does not appear to be installed as a service (localhost port 9200)'
pytest.exit(msg)
```
- Note that the single argument of `pytest_configure` has to be named `config`.
- Using `pytest.exit` makes it look nicer.
feat(python_docker#Using PDM): Dockerize a PDM application
It is possible to use PDM in a multi-stage Dockerfile to first install the project and dependencies into `__pypackages__` and then copy this folder into the final stage, adding it to `PYTHONPATH`.
```dockerfile
FROM python:3.11-slim-bookworm AS builder
RUN pip install pdm
COPY pyproject.toml pdm.lock README.md /project/
COPY src/ /project/src
WORKDIR /project
RUN mkdir __pypackages__ && pdm sync --prod --no-editable
FROM python:3.11-slim-bookworm
ENV PYTHONPATH=/project/pkgs
COPY --from=builder /project/__pypackages__/3.11/lib /project/pkgs
COPY --from=builder /project/__pypackages__/3.11/bin/* /bin/
CMD ["python", "-m", "project"]
```
feat(python_snippets#Configure the logging of a program to look nice): Configure the logging of a program to look nice
```python
def load_logger(verbose: bool = False) -> None: # pragma no cover
"""Configure the Logging logger.
Args:
verbose: Set the logging level to Debug.
"""
logging.addLevelName(logging.INFO, "\033[36mINFO\033[0m")
logging.addLevelName(logging.ERROR, "\033[31mERROR\033[0m")
logging.addLevelName(logging.DEBUG, "\033[32mDEBUG\033[0m")
logging.addLevelName(logging.WARNING, "\033[33mWARNING\033[0m")
if verbose:
logging.basicConfig(
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
stream=sys.stderr,
level=logging.DEBUG,
datefmt="%Y-%m-%d %H:%M:%S",
)
telebot.logger.setLevel(logging.DEBUG) # Outputs debug messages to console.
else:
logging.basicConfig(
stream=sys.stderr, level=logging.INFO, format="%(levelname)s: %(message)s"
)
```
feat(python_snippets#Get the modified time of a file with Pathlib): Get the modified time of a file with Pathlib
```python
file_ = Path('/to/some/file')
file_.stat().st_mtime
```
You can also access:
- Created time: with `st_ctime`
- Accessed time: with `st_atime`
They are timestamps, so if you want to compare it with a datetime object use the `timestamp` method:
```python
assert datetime.now().timestamp - file_.stat().st_mtime < 60
```
feat(collaborating_tools): Introduce collaborating tools
Collaborating document creation:
- https://pad.riseup.net
- https://rustpad.io . [Can be self hosted](https://github.com/ekzhang/rustpad)
Collaborating through terminals:
- [sshx](https://sshx.io/) looks promising although I think it uses their servers to do the connection, which is troublesome.
fix(kubernetes_tools#Tried): Recommend rke2 over k3s
A friend told me that it works better.
feat(emojis#Most used): Create a list of most used emojis
```
¯\(°_o)/¯
¯\_(ツ)_/¯
(╯°□°)╯ ┻━┻
＼\ ٩( ᐛ )و /／
(✿◠‿◠)
(/ﾟДﾟ)/
(¬º-°)¬
(╥﹏╥)
ᕕ( ᐛ )ᕗ
ʕ•ᴥ•ʔ
( ˘ ³˘)♥
❤
```
feat(gitea#Run jobs if other jobs failed): Run jobs if other jobs failed
This is useful to send notifications if any of the jobs failed.
[Right now](go-gitea/gitea#23725) you can't run a job if other jobs fail, all you can do is add a last step on each workflow to do the notification on failure:
```yaml
- name: Send mail
if: failure()
uses: https://github.com/dawidd6/action-send-mail@v3
with:
to: ${{ secrets.MAIL_TO }}
from: Gitea <gitea@hostname>
subject: ${{ gitea.repository }} ${{gitea.workflow}} ${{ job.status }}
priority: high
convert_markdown: true
html_body: |
### Job ${{ job.status }}
${{ github.repository }}: [${{ github.ref }}@${{ github.sha }}](${{ github.server_url }}/${{ github.repository }}/actions)
```
feat(grapheneos#Split the screen): Split the screen
Go into app switcher, tap on the app icon above the active app and then select "Split top".
feat(how_to_code): Personal evolution on how I code
Over the years I've tried different ways of developing my code:
- Mindless coding: write code as you need to make it work, with no tests, documentation or any quality measure.
- TDD.
- Try to abstract everything to minimize the duplication of code between projects.
Each has it's advantages and disadvantages. After trying them all and given that right now I only have short spikes of energy and time to invest in coding my plan is to:
- Make the minimum effort to design the minimum program able to solve the problem at hand. This design will be represented in an [orgmode](orgmode.md) task.
- Write the minimum code to make it work without thinking of tests or generalization, but with the [domain driven design](domain_driven_design.md) concepts so the code remains flexible and maintainable.
- Once it's working see if I have time to improve it:
- Create the tests to cover the critical functionality (no more 100% coverage).
- If I need to make a package or the program evolves into something complex I'd use [this scaffold template](https://github.com/lyz-code/cookiecutter-python-project).
Once the spike is over I'll wait for a new spike to come either because I have time or because something breaks and I need to fix it.
feat(life_analysis): Introduce the analysis of life process
It's interesting to do analysis at representative moments of the year. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:
- Spring analysis (1st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.
- Summer analysis (1st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop *easy* and *chill* personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.
- Autumn analysis (1st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.
- Winter analysis (1st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year.
We see then that the year is divided in two sets of an expansion trimester and a retreat one. We can use this information to plan our tasks accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.
feat(life_planning#month-plan): Introduce the month planning process
The objectives of the month plan are:
- Define the month objectives according to the trimester plan and the insights gathered in the past month review.
- Make your backlog and todo list match the month objectives
- Define the philosophical topics to address
- Define the topics to learn
- Define the are of habits to incorporate?
- Define the checks you want to do at the end of the month.
- Plan when is it going to be the next review.
It's interesting to do the plannings on meaningful days such as the first one of the month. Usually we don't have enough flexibility in our life to do it exactly that day, so schedule it the closest you can to that date. It's a good idea to do both the review and the planning on the same day.
We'll divide the planning process in these phases:
- Prepare
- Clarify your state
- Decide the month objectives
Prepare:
It's important that you prepare your environment for the planning. You need to be present and fully focused on the process itself. To do so you can:
- Make sure you don't get interrupted:
- Check your task manager tools to make sure that you don't have anything urgent to address in the next hour.
- Disable all notifications
- Set your analysis environment:
- Put on the music that helps you get *in the zone*.
- Get all the things you may need for the review:
- The checklist that defines the process of your planning (this document in my case).
- Somewhere to write down the insights.
- Your task manager system
- Your habit manager system
- Your *Objective list*.
- Your *Thinking list*.
- Your *Reading list*.
- Remove from your environment everything else that may distract you
Clarify your state:
To be able to make a good decision on your month's path you need to sort out which is your current state. To do so:
- Clean your inbox: Refile each item until it's empty
- Clean your todo: Review each todo element by deciding if they should still be in the todo. If they do and they belong to a month objective, add it. If they don't need to be in the todo, refile it.
- Clean your someday: Review each relevant someday element (not the ones that are archive at greater levels than month) and decide if they should be refiled elsewhere and if they are part of a month objective that should be dealt with this month.
- Adress each of the trimester objectives by creating month objectives that get you closer to the desired objective.
Decide the next steps:
For each of your month objectives:
- Decide wheter it makes sense to address it this month. If not, archive it
- Create a clear plan of action for this month on that objective
- Tweak your *things to think about list*.
- Tweak your *reading list*.
- Tweak your *habit manager system*.
feat(linux_snippets#Accept new ssh keys by default): Accept new ssh keys by default
While common wisdom is not to disable host key checking, there is a built-in option in SSH itself to do this. It is relatively unknown, since it's new (added in Openssh 6.5).
This is done with `-o StrictHostKeyChecking=accept-new`. Or if you want to use it for all hosts you can add the next lines to your `~/.ssh/config`:
```
Host *
StrictHostKeyChecking accept-new
```
WARNING: use this only if you absolutely trust the IP\hostname you are going to SSH to:
```bash
ssh -o StrictHostKeyChecking=accept-new mynewserver.example.com
```
Note, `StrictHostKeyChecking=no` will add the public key to `~/.ssh/known_hosts` even if the key was changed. `accept-new` is only for new hosts. From the man page:
> If this flag is set to “accept-new” then ssh will automatically add new host keys to the user known hosts files, but will not permit connections to hosts with changed host keys. If this flag is set to “no” or “off”, ssh will automatically add new host keys to the user known hosts files and allow connections to hosts with changed hostkeys to proceed, subject to some restrictions. If this flag is set to ask (the default), new host keys will be added to the user known host files only after the user has confirmed that is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed. The host keys of known hosts will be verified automatically in all cases.
feat(linux_snippets#Do not add trailing / to ls): Do not add trailing / to ls
Probably, your `ls` is aliased or defined as a function in your config files.
Use the full path to `ls` like:
```bash
/bin/ls /var/lib/mysql/
```
feat(linux_snippets#Convert png to svg): Convert png to svg
Inkscape has got an awesome auto-tracing tool.
- Install Inkscape using `sudo apt-get install inkscape`
- Import your image
- Select your image
- From the menu bar, select Path > Trace Bitmap Item
- Adjust the tracing parameters as needed
- Save as svg
Check their [tracing tutorial](https://inkscape.org/en/doc/tutorials/tracing/tutorial-tracing.html) for more information.
Once you are comfortable with the tracing options. You can automate it by using [CLI of Inkscape](https://inkscape.org/en/doc/inkscape-man.html).
feat(linux_snippets#Redirect stdout and stderr of a cron job to a file): Redirect stdout and stderr of a cron job to a file
```
*/1 * * * * /home/ranveer/vimbackup.sh >> /home/ranveer/vimbackup.log 2>&1
```
feat(linux_snippets#Error when unmounting a device Target is busy): Error when unmounting a device Target is busy
- Check the processes that are using the mountpoint with `lsof /path/to/mountpoint`
- Kill those processes
- Try the umount again
If that fails, you can use `umount -l`.
feat(loki#installation): How to install loki
There are [many ways to install Loki](https://grafana.com/docs/loki/latest/setup/install/), we're going to do it using `docker-compose` taking [their example as a starting point](https://raw.githubusercontent.com/grafana/loki/v2.9.1/production/docker-compose.yaml) and complementing our already existent [grafana docker-compose](grafana.md#installation).
It makes use of the [environment variables to configure Loki](https://grafana.com/docs/loki/latest/configure/#configuration-file-reference), that's why we have the `-config.expand-env=true` flag in the command line launch.
In the grafana datasources directory add `loki.yaml`:
```yaml
---
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
basicAuth: false
isDefault: true
version: 1
editable: false
```
[Storage configuration](https://grafana.com/docs/loki/latest/storage/):
Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.
Loki 2.0 brings an index mechanism named ‘boltdb-shipper’ and is what we now call Single Store. This type only requires one store, the object store, for both the index and chunks.
Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki as it improves query performance, reduces TCO and has the same feature parity as “boltdb-shipper”.
feat(orgzly#Avoid the conflicts in the files edited in two places): Avoid the conflicts in the files edited in two places
If you use syncthing you may be seeing conflicts in your files. This happens specially if you use the Orgzly widget to add tasks, this is because it doesn't synchronize the files to the directory when using the widget. If you have a file that changes a lot in a device, for example the `inbox.org` of my mobile, it's interesting to have a specific file that's edited mainly in the mobile, and when you want to edit it elsewhere, you sync as specified below and then process with the editing. Once it's done manually sync the changes in orgzly again. The rest of the files synced to the mobile are for read only reference, so they rarely change.
If you want to sync reducing the chance of conflicts then:
- Open Orgzly and press Synchronize
- Open Syncthing.
If that's not enough [check these automated solutions](orgzly/orgzly-android#8):
- [Orgzly auto syncronisation for sync tools like syncthing](https://gist.github.com/fabian-thomas/6f559d0b0d26737cf173e41cdae5bfc8)
- [watch-for-orgzly](https://gitlab.com/doak/orgzly-watcher/-/blob/master/watch-for-orgzly?ref_type=heads)
Other interesting solutions:
- [org-orgzly](https://codeberg.org/anoduck/org-orgzly): Script to parse a chosen org file or files, check if an entry meets required parameters, and if it does, write the entry in a new file located inside the folder you desire to sync with orgzly.
- [Git synchronization](orgzly/orgzly-android#24): I find it more cumbersome than syncthing but maybe it's interesting for you.
feat(orgzly#references): add new orgzly fork
[Alternative fork maintained by the community](https://github.com/orgzly-revived/orgzly-android-revived)
feat(pytelegrambotapi): Introduce pytelegrambotapi
[pyTelegramBotAPI](https://github.com/eternnoir/pyTelegramBotAPI) is an synchronous and asynchronous implementation of the [Telegram Bot API](https://core.telegram.org/bots/api).
[Installation](https://pytba.readthedocs.io/en/latest/install.html):
```bash
pip install pyTelegramBotAPI
```
feat(pytelegrambotapi#Create your bot): Create your bot
Use the `/newbot` command to create a new bot. `@BotFather` will ask you for a name and username, then generate an authentication token for your new bot.
- The `name` of your bot is displayed in contact details and elsewhere.
- The `username` is a short name, used in search, mentions and t.me links. Usernames are 5-32 characters long and not case sensitive – but may only include Latin characters, numbers, and underscores. Your bot's username must end in 'bot’, like `tetris_bot` or `TetrisBot`.
- The `token` is a string, like `110201543:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw`, which is required to authorize the bot and send requests to the Bot API. Keep your token secure and store it safely, it can be used by anyone to control your bot.
To edit your bot, you have the next available commands:
- `/setname`: change your bot's name.
- `/setdescription`: change the bot's description (short text up to 512 characters). Users will see this text at the beginning of the conversation with the bot, titled 'What can this bot do?'.
- `/setabouttext`: change the bot's about info, a shorter text up to 120 characters. Users will see this text on the bot's profile page. When they share your bot with someone, this text is sent together with the link.
- `/setuserpic`: change the bot's profile picture.
- `/setcommands`: change the list of commands supported by your bot. Users will see these commands as suggestions when they type / in the chat with your bot. See commands for more info.
- `/setdomain`: link a website domain to your bot. See the login widget section.
- `/deletebot`: delete your bot and free its username. Cannot be undone.
feat(pytelegrambotapi#Synchronous TeleBot): Synchronous TeleBot
```python
import telebot
API_TOKEN = '<api_token>'
bot = telebot.TeleBot(API_TOKEN)
@bot.message_handler(commands=['help', 'start'])
def send_welcome(message):
bot.reply_to(message, """\
Hi there, I am EchoBot.
I am here to echo your kind words back to you. Just say anything nice and I'll say the exact same thing to you!\
""")
@bot.message_handler(func=lambda message: True)
def echo_message(message):
bot.reply_to(message, message.text)
bot.infinity_polling()
```
feat(pytelegrambotapi#Asynchronous TeleBot): Asynchronous TeleBot
```python
from telebot.async_telebot import AsyncTeleBot
bot = AsyncTeleBot('TOKEN')
@bot.message_handler(commands=['help', 'start'])
async def send_welcome(message):
await bot.reply_to(message, """\
Hi there, I am EchoBot.
I am here to echo your kind words back to you. Just say anything nice and I'll say the exact same thing to you!\
""")
@bot.message_handler(func=lambda message: True)
async def echo_message(message):
await bot.reply_to(message, message.text)
import asyncio
asyncio.run(bot.polling())
```
feat(pytest-xprocess): Introduce pytest-xprocess
[`pytest-xprocess`](https://github.com/pytest-dev/pytest-xprocess) is a pytest plugin for managing external processes across test runs.
[Installation](https://pytest-xprocess.readthedocs.io/en/latest/#quickstart):
```bash
pip install pytest-xprocess
```
[Usage](https://pytest-xprocess.readthedocs.io/en/latest/#quickstart):
Define your process fixture in `conftest.py`:
```python
import pytest
from xprocess import ProcessStarter
@pytest.fixture
def myserver(xprocess):
class Starter(ProcessStarter):
# startup pattern
pattern = "[Ss]erver has started!"
# command to start process
args = ['command', 'arg1', 'arg2']
# ensure process is running and return its logfile
logfile = xprocess.ensure("myserver", Starter)
conn = # create a connection or url/port info to the server
yield conn
# clean up whole process tree afterwards
xprocess.getinfo("myserver").terminate()
```
Now you can use this fixture in any test functions where `myserver` needs to be up and `xprocess` will take care of it for you.
[Matching process output with pattern](https://pytest-xprocess.readthedocs.io/en/latest/starter.html#matching-process-output-with-pattern):
In order to detect that your process is ready to answer queries,
`pytest-xprocess` allows the user to provide a string pattern by setting the
class variable pattern in the Starter class. `pattern` will be waited for in
the process `logfile` for a maximum time defined by `timeout` before timing out in
case the provided pattern is not matched.
It’s important to note that pattern is a regular expression and will be matched using python `re.search`.
[Controlling Startup Wait Time with timeout](https://pytest-xprocess.readthedocs.io/en/latest/starter.html#controlling-startup-wait-time-with-timeout):
Some processes naturally take longer to start than others. By default,
`pytest-xprocess` will wait for a maximum of 120 seconds for a given process to
start before raising a `TimeoutError`. Changing this value may be useful, for
example, when the user knows that a given process would never take longer than
a known amount of time to start under normal circunstancies, so if it does go
over this known upper boundary, that means something is wrong and the waiting
process must be interrupted. The maximum wait time can be controlled through the
class variable timeout.
```python
@pytest.fixture
def myserver(xprocess):
class Starter(ProcessStarter):
# will wait for 10 seconds before timing out
timeout = 10
```
Passing command line arguments to your process with `args`:
In order to start a process, pytest-xprocess must be given a command to be passed into the subprocess.Popen constructor. Any arguments passed to the process command can also be passed using args. As an example, if I usually use the following command to start a given process:
```bash
$> myproc -name "bacon" -cores 4 <destdir>
```
That would look like:
```python
args = ['myproc', '-name', '"bacon"', '-cores', 4, '<destdir>']
```
when using args in pytest-xprocess to start the same process.
```python
@pytest.fixture
def myserver(xprocess):
class Starter(ProcessStarter):
# will pass "$> myproc -name "bacon" -cores 4 <destdir>" to the
# subprocess.Popen constructor so the process can be started with
# the given arguments
args = ['myproc', '-name', '"bacon"', '-cores', 4, '<destdir>']
# ...
```
feat(python_prometheus): How to create a prometheus exporter with python
[prometheus-client](https://github.com/prometheus/client_python) is the official Python client for [Prometheus](prometheus.md).
Installation:
```bash
pip install prometheus-client
```
Here is a simple script:
```python
from prometheus_client import start_http_server, Summary
import random
import time
REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
@REQUEST_TIME.time()
def process_request(t):
"""A dummy function that takes some time."""
time.sleep(t)
if __name__ == '__main__':
# Start up the server to expose the metrics.
start_http_server(8000)
# Generate some requests.
while True:
process_request(random.random())
```
Then you can visit http://localhost:8000/ to view the metrics.
From one easy to use decorator you get:
- `request_processing_seconds_count`: Number of times this function was called.
- `request_processing_seconds_sum`: Total amount of time spent in this function.
Prometheus's rate function allows calculation of both requests per second, and latency over time from this data.
In addition if you're on Linux the process metrics expose CPU, memory and other information about the process for free.
feat(python-telegram): Analyze the different python libraries to interact with telegram
There are two ways to interact with Telegram through python:
- Client libraries
- Bot libraries
Client libraries:
Client libraries use your account to interact with Telegram itself through a developer API token.
The most popular to use is [Telethon](https://docs.telethon.dev/en/stable/index.html).
Bot libraries:
[Telegram lists many libraries to interact with the bot API](https://core.telegram.org/bots/samples#python), the most interesting are:
- [python-telegram-bot](#python-telegram-bot)
- [pyTelegramBotAPI](#pytelegrambotapi)
- [aiogram](#aiogram)
If there comes a moment when we have to create the messages ourselves, [telegram-text](https://telegram-text.alinsky.tech/api_reference) may be an interesting library to check.
[python-telegram-bot](https://github.com/python-telegram-bot/python-telegram-bot):
Pros:
- Popular: 23k stars, 4.9k forks
- Maintained: last commit 3 days ago
- They have a developers community to get help in [this telegram group](https://telegram.me/pythontelegrambotgroup)
- I like how they try to minimize third party dependencies, and how you can install the complements if you need them
- Built on top of asyncio
- Nice docs
- Fully supports the [Telegram bot API](https://core.telegram.org/bots/api)
- Has many examples
Cons:
- Interface is a little verbose and complicated at a first look
- Only to be run in a single thread (not a problem)
References:
- [Package documentation](https://docs.python-telegram-bot.org/) is the technical reference for python-telegram-bot. It contains descriptions of all available classes, modules, methods and arguments as well as the changelog.
- [Wiki](https://github.com/python-telegram-bot/python-telegram-bot/wiki/) is home to number of more elaborate introductions of the different features of python-telegram-bot and other useful resources that go beyond the technical documentation.
- [Examples](https://docs.python-telegram-bot.org/examples.html) section contains several examples that showcase the different features of both the Bot API and python-telegram-bot
- [Source](https://github.com/python-telegram-bot/python-telegram-bot)
[pyTelegramBotAPI](https://github.com/eternnoir/pyTelegramBotAPI):
Pros:
- Popular: 7.1k stars, 1.8k forks
- Maintained: last commit 3 weeks ago
- Both sync and async
- Nicer interface with decorators and simpler setup
- [They have an example on how to split long messages](https://github.com/eternnoir/pyTelegramBotAPI#sending-large-text-messages)
- [Nice docs on how to test](https://github.com/eternnoir/pyTelegramBotAPI#testing)
- They have a developers community to get help in [this telegram group](https://telegram.me/joinchat/Bn4ixj84FIZVkwhk2jag6A)
- Fully supports the [Telegram bot API](https://core.telegram.org/bots/api)
- Has examples
Cons:
- Uses lambdas inside the decorators, I don't know why it does it.
- The docs are not as throughout as `python-telegram-bot` one.
References:
- [Documentation](https://pytba.readthedocs.io/en/latest/index.html)
- [Source](https://github.com/eternnoir/pyTelegramBotAPI)
- [Async Examples](https://github.com/eternnoir/pyTelegramBotAPI/tree/master/examples/asynchronous_telebot)
[aiogram](https://github.com/aiogram/aiogram):
Pros:
- Popular: 3.8k stars, 717k forks
- Maintained: last commit 4 days ago
- Async support
- They have a developers community to get help in [this telegram group](https://t.me/aiogram)
- Has type hints
- Cleaner interface than `python-telegram-bot`
- Fully supports the [Telegram bot API](https://core.telegram.org/bots/api)
- Has examples
Cons:
- Less popular than `python-telegram-bot`
- Docs are written at a developer level, difficult initial barrier to understand how to use it.
References:
- [Documentation](https://docs.aiogram.dev/en/dev-3.x/)
- [Source](https://github.com/aiogram/aiogram)
- [Examples](https://github.com/aiogram/aiogram/tree/dev-3.x/examples)
Conclusion:
Even if `python-telegram-bot` is the most popular and with the best docs, I prefer one of the others due to the easier interface. `aiogram`s documentation is kind of crap, and as it's the first time I make a bot I'd rather have somewhere good to look at.
So I'd say to go first with `pyTelegramBotAPI` and if it doesn't go well, fall back to `python-telegram-bot`.
feat(rocketchat): Introduce Rocketchat integrations
Rocket.Chat supports webhooks to integrate tools and services you like into the platform. Webhooks are simple event notifications via HTTP POST. This way, any webhook application can post a message to a Rocket.Chat instance and much more.
With scripts, you can point any webhook to Rocket.Chat and process the requests to print customized messages, define the username and avatar of the user of the messages and change the channel for sending messages, or you can cancel the request to prevent undesired messages.
Available integrations:
- Incoming Webhook: Let an external service send a request to Rocket.Chat to be processed.
- Outgoing Webhook: Let Rocket.Chat trigger and optionally send a request to an external service and process the response.
By default, a webhook is designed to post messages only. The message is part of a JSON structure, which has the same format as that of a .
[Incoming webhook script](https://docs.rocket.chat/use-rocket.chat/workspace-administration/integrations#incoming-webhook-script):
To create a new incoming webhook:
- Navigate to Administration > Workspace > Integrations.
- Click +New at the top right corner.
- Switch to the Incoming tab.
- Turn on the Enabled toggle.
- Name: Enter a name for your webhook. The name is optional; however, providing a name to manage your integrations easily is advisable.
- Post to Channel: Select the channel (or user) where you prefer to receive the alerts. It is possible to override messages.
- Post as: Choose the username that this integration posts as. The user must already exist.
- Alias: Optionally enter a nickname that appears before the username in messages.
- Avatar URL: Enter a link to an image as the avatar URL if you have one. The avatar URL overrides the default avatar.
- Emoji: Enter an emoji optionally to use the emoji as the avatar. [Check the emoji cheat sheet](https://github.com/ikatyang/emoji-cheat-sheet/blob/master/README.md#computer)
- Turn on the Script Enabled toggle.
- Paste your script inside the Script field (check below for a sample script)
- Save the integration.
- Use the generated Webhook URL to post messages to Rocket.Chat.
The Rocket.Chat integration script should be written in ES2015 / ECMAScript 6. The script requires a global class named Script, which is instantiated only once during the first execution and kept in memory. This class contains a method called `process_incoming_request`, which is called by your server each time it receives a new request. The `process_incoming_request` method takes an object as a parameter with the request property and returns an object with a content property containing a valid Rocket.Chat message, or an object with an error property, which is returned as the response to the request in JSON format with a Code 400 status.
A valid Rocket.Chat message must contain a text field that serves as the body of the message. If you redirect the message to a channel other than the one indicated by the webhook token, you can specify a channel field that accepts room id or, if prefixed with "#" or "@", channel name or user, respectively.
You can use the console methods to log information to help debug your script. More information about the console can be found [here](https://developer.mozilla.org/en-US/docs/Web/API/Console/log).
. To view the logs, navigate to Administration > Workspace > View Logs.
```
/* exported Script */
/* globals console, _, s */
/** Global Helpers
*
* console - A normal console instance
* _ - An underscore instance
* s - An underscore string instance
*/
class Script {
/**
* @params {object} request
*/
process_incoming_request({ request }) {
// request.url.hash
// request.url.search
// request.url.query
// request.url.pathname
// request.url.path
// request.url_raw
// request.url_params
// request.headers
// request.user._id
// request.user.name
// request.user.username
// request.content_raw
// request.content
// console is a global helper to improve debug
console.log(request.content);
return {
content:{
text: request.content.text,
icon_emoji: request.content.icon_emoji,
channel: request.content.channel,
// "attachments": [{
// "color": "#FF0000",
// "author_name": "Rocket.Cat",
// "author_link": "https://open.rocket.chat/direct/rocket.cat",
// "author_icon": "https://open.rocket.chat/avatar/rocket.cat.jpg",
// "title": "Rocket.Chat",
// "title_link": "https://rocket.chat",
// "text": "Rocket.Chat, the best open source chat",
// "fields": [{
// "title": "Priority",
// "value": "High",
// "short": false
// }],
// "image_url": "https://rocket.chat/images/mockup.png",
// "thumb_url": "https://rocket.chat/images/mockup.png"
// }]
}
};
// return {
// error: {
// success: false,
// message: 'Error example'
// }
// };
}
}
```
To test if your integration works, use curl to make a POST request to the generated webhook URL.
```bash
curl -X POST \
-H 'Content-Type: application/json' \
--data '{
"icon_emoji": ":smirk:",
"text": "Example message"
}' \
https://your-webhook-url
```
If you want to send the message to another channel or user use the `channel` argument with `@user` or `#channel`. Keep in mind that the user of the integration needs to be part of those channels if they are private.
```bash
curl -X POST \
-H 'Content-Type: application/json' \
--data '{
"icon_emoji": ":smirk:",
"channel": "#notifications",
"text": "Example message"
}' \
https://your-webhook-url
```
If you want to do more complex things uncomment the part of the attachments.
feat(siem): Add Wazuh SIEM
[Wazuh](https://wazuh.com/)
feat(tails): Add interesting operations on tails
- [Upgrading a tails USB](https://tails.net/upgrade/tails/index.en.html)
- [Change the window manager](https://www.reddit.com/r/tails/comments/qzruhv/changing_window_manager/): Don't do it, they say it it will break Tails although I don't understand why
feat(vim#Email inside nvim): Email inside nvim
The best looking one is himalaya
- [Home](https://pimalaya.org/himalaya/index.html)
- [Nvim plugin](https://git.sr.ht/%7Esoywod/himalaya-vim)
- [Source](https://github.com/soywod/himalaya)
and others
The possibility of new synchroniztion methods was mentioned way back in 2015 https://plus.google.com/+Orgzly/posts/S2nBbxVibsB
Did anything become of this? Are there plans to add git synchronization support to orgzly?
The text was updated successfully, but these errors were encountered: