Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Livelabs ID: 11105] Story of a Hack updates #278

Merged
merged 456 commits into from
Dec 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
456 commits
Select commit Hold shift + click to select a range
60a3ecc
dbseclab_v5.5
Dec 1, 2023
c3461e8
dbseclab_v5.5
Dec 1, 2023
a456bb6
dbseclab_v5.5
Dec 1, 2023
52f7273
dbseclab_v5.5
Dec 1, 2023
94a18ec
dbseclab_v5.5
Dec 1, 2023
b5d31d2
dbseclab_v5.5
Dec 1, 2023
737b1c4
dbseclab_v5.5
Dec 1, 2023
7ce8c80
dbseclab_v5.5
Dec 1, 2023
3730fd6
dbseclab_v5.5
Dec 1, 2023
8186c30
dbseclab_v5.5
Dec 1, 2023
46352cf
dbseclab_v5.5
Dec 1, 2023
ab86451
Merge remote-tracking branch 'upstream/main'
Dec 21, 2023
e835fd9
dbseclab_v5.5
Dec 21, 2023
e768513
Merge remote-tracking branch 'upstream/main'
Dec 21, 2023
1d81753
dbseclab_v5.5
Dec 21, 2023
8fd4219
Squashed commit of the following:
Dec 21, 2023
d44dabe
dbseclab_v5.5
Dec 21, 2023
a0d9102
Merge remote-tracking branch 'upstream/main'
Dec 21, 2023
ff57e25
Merge remote-tracking branch 'upstream/main'
Dec 21, 2023
c00b2cf
dbseclab_v5
Jan 3, 2024
ad0ac6a
dbseclab_v5.5
Jan 3, 2024
6be94b6
dbseclab_v5.6
Jan 11, 2024
a28e166
dbseclab_v5.6
Jan 16, 2024
ec8c74b
Merge remote-tracking branch 'upstream/main'
Jan 16, 2024
3f67d64
dbseclab_v5.6
Jan 19, 2024
b286bc7
dbseclab_v6.0
Feb 8, 2024
257d17c
dbseclab_v6.0
Feb 9, 2024
98caeea
dbseclab_v6.0
Feb 9, 2024
9765161
dbseclab_v6.0
Feb 28, 2024
630d452
dbseclab_v6.0
Feb 28, 2024
8f0ea86
Merge remote-tracking branch 'upstream/main'
Mar 1, 2024
75153f0
dbseclab_v6.0
Mar 1, 2024
db7a5cb
dbseclab_v6.0
Mar 1, 2024
5bcd236
dbseclab_v6.0
Mar 7, 2024
30d28dc
dbseclab_v6.0
Mar 12, 2024
1f356bc
Merge remote-tracking branch 'upstream/main'
Mar 13, 2024
f3ffe80
Merge remote-tracking branch 'upstream/main'
Mar 26, 2024
11ca879
dbseclab_v6.0
Mar 26, 2024
0ddc337
Merge remote-tracking branch 'upstream/main'
Apr 3, 2024
1929e0b
dbseclab-v6.1
Apr 3, 2024
83a18b5
dbseclab-v6.1
Apr 3, 2024
843af16
dbseclab-v6.1
Apr 3, 2024
412f6af
dbseclab-v6.1
Apr 3, 2024
c7cdef1
dbseclab-v6.1
Apr 4, 2024
0f6a510
dbseclab_v6.1
Apr 18, 2024
b873528
dbseclab-v6.1
Apr 18, 2024
33ab1c3
dbseclab-v6.1
Apr 20, 2024
fb4d7fc
Merge remote-tracking branch 'upstream/main'
Apr 20, 2024
060a88f
dbseclab-v6.1
Apr 25, 2024
eeb178c
Merge remote-tracking branch 'upstream/main'
Apr 25, 2024
a505a63
dbseclab-v6.1
Apr 25, 2024
3837293
dbseclab-v6.1
Apr 25, 2024
457a425
dbseclab-v6.1
Apr 25, 2024
d6ce4c2
Merge remote-tracking branch 'upstream/main'
Apr 25, 2024
f6726fc
dbseclab-v6.1
Apr 26, 2024
2c65bd4
dbseclab-v6.1
Apr 29, 2024
30dfb3f
Merge remote-tracking branch 'upstream/main'
Apr 29, 2024
889ebfd
dbseclab-v6.1
Apr 30, 2024
dd4ac25
dbseclab-v6.2
Apr 30, 2024
7175275
dbseclab-v6.2
May 2, 2024
12cdd6c
dbseclab-v6.1
May 2, 2024
3206340
dbseclab-v6.1
May 2, 2024
9c0892f
Merge remote-tracking branch 'upstream/main'
May 3, 2024
c332812
dbseclab-v6.1
May 3, 2024
2575f64
dbseclab-v6.1
May 7, 2024
8f4f180
Merge remote-tracking branch 'upstream/main'
May 7, 2024
8d55c1e
dbseclab-v6.1
May 7, 2024
f917d32
updating dv lab - rce
richardcevans May 7, 2024
f31dbe1
small updates - rce
richardcevans May 7, 2024
956ee0e
make changes - rce
richardcevans May 7, 2024
3bf7a84
dbseclab-v6.1
May 7, 2024
494cdf7
dbseclab-v6.1
May 7, 2024
21efe12
Merge remote-tracking branch 'upstream/main'
May 7, 2024
fc6018a
dbseclab-v6.1
May 14, 2024
13076e0
Merge remote-tracking branch 'upstream/main'
May 14, 2024
da1d577
dbseclab-v6.1
May 14, 2024
30af993
Merge remote-tracking branch 'upstream/main'
May 16, 2024
cf7172d
dbseclab-v6.2
May 16, 2024
d43b7ad
dbseclab-v6.2
May 16, 2024
9393671
Updates for 23ai
richardcevans May 20, 2024
48a7e90
Updates for labs
richardcevans May 22, 2024
fe9d3b2
Merge branch 'main' of https://github.com/hloumi/security
richardcevans May 22, 2024
c3a6bf9
update dv labs
richardcevans May 22, 2024
3b06d0d
updates to the lab
richardcevans May 22, 2024
55aff58
dv lab updates
richardcevans May 22, 2024
1211a0e
dbseclab_v70
May 22, 2024
895d611
dbseclab-v6.2
May 23, 2024
a810bfd
lab updates
richardcevans May 23, 2024
437c8cc
Merge branch 'main' of https://github.com/hloumi/security
richardcevans May 23, 2024
d7869ae
update lab
richardcevans May 23, 2024
720bb6f
updates to adb dv lab
richardcevans May 23, 2024
545e011
adb dv lab updates
richardcevans May 23, 2024
5e60c03
adb dv lab updates
richardcevans May 23, 2024
ab62abd
adb dv lab updates
richardcevans May 23, 2024
e032d8b
dbseclab-v6.2
May 23, 2024
00e4379
Merge branch 'main' of https://github.com/hloumi/security
May 23, 2024
076f4ad
Merge remote-tracking branch 'upstream/main'
May 23, 2024
02942a8
dbseclab-v6.2
May 23, 2024
5cd52e8
adb dv lab updates
richardcevans May 23, 2024
b7a43e8
dbseclab-v6.2
May 23, 2024
d969429
dbseclab-v6.2
May 23, 2024
21bc41e
Merge branch 'main' of https://github.com/hloumi/security
richardcevans May 23, 2024
28d99ef
dbseclab-v6.2
May 23, 2024
e99ddf0
Merge branch 'main' of https://github.com/hloumi/security
May 23, 2024
0fceaf9
adb dv lab updates
richardcevans May 23, 2024
141ba91
adb dbv lab updates
richardcevans May 23, 2024
e397aba
dbseclab-v6.2
May 24, 2024
5fc501a
dbseclab-v6.2
May 24, 2024
58b92b7
dbseclab-v6.2
May 24, 2024
f9b908d
dbseclab-v6.2
May 24, 2024
8028048
dbseclab-v6.2
May 24, 2024
604128d
dbseclab-v6.2
May 24, 2024
a3243c8
updates to adb dv lab
richardcevans May 26, 2024
427b6da
adb dv lab
richardcevans May 26, 2024
9f51b53
adb dv labs
richardcevans May 26, 2024
ba2a6df
dbseclab-v6.2
May 27, 2024
5d319ff
Merge branch 'main' of https://github.com/hloumi/security
May 27, 2024
cfe7d11
dbseclab-v6.2
May 27, 2024
6755918
dbseclab-v6.2
May 27, 2024
72e2640
dbseclab-v6.2
May 27, 2024
b21787a
dbseclab-v6.2
May 27, 2024
7168527
dbseclab-v6.2
May 28, 2024
f8a6d48
dbseclab-v6.2
May 28, 2024
9a5ff31
dbseclab-v6.2
May 28, 2024
43aa4b4
dbseclab-v6.2
May 28, 2024
30d4eb2
dbseclab-v6.2
May 28, 2024
afc66d8
dbseclab-v6.2
May 28, 2024
91608fa
dv lab updates
richardcevans May 28, 2024
d027d40
adb dv lab updates
richardcevans May 28, 2024
967deb9
adb dv lab updates
richardcevans May 28, 2024
4cf3a88
adb dv lab update
richardcevans May 28, 2024
c5a20cb
adb dv
richardcevans May 28, 2024
0d2c790
adb dv labs
richardcevans May 28, 2024
0e59538
adb dv lab updates
richardcevans May 28, 2024
078fafb
dv lab updates
richardcevans May 29, 2024
72284d6
dv labs update
richardcevans May 29, 2024
a1e5f12
dbseclab-v6.2
May 29, 2024
867924f
Merge branch 'main' of https://github.com/hloumi/security
May 29, 2024
7307cc9
Merge remote-tracking branch 'upstream/main'
May 29, 2024
3a1f72e
dbseclab-v6.2
May 31, 2024
627b5ee
dbseclab-v6.2
Jun 3, 2024
5c5f681
Merge remote-tracking branch 'upstream/main'
Jun 3, 2024
50ead4d
dbseclab-v6.2
Jun 3, 2024
fb00f9d
dbseclab-v6.2
Jun 6, 2024
ed3c2da
dbseclab-v6.2
Jun 7, 2024
d825117
dbseclab-v6.2
Jun 7, 2024
47336d8
dbseclab-v6.2
Jun 7, 2024
8cf77ca
dbseclab-v6.2
Jun 7, 2024
d1217b4
dbseclab-v6.2
Jun 7, 2024
4a1c810
dbseclab-v6.2
Jun 7, 2024
d46cb77
dbseclab-v6.2
Jun 7, 2024
6dde443
dbseclab-v6.2
Jun 7, 2024
47f24d8
dbseclab-v6.2
Jun 7, 2024
0b62949
dbseclab-v6.2
Jun 7, 2024
3d8aa73
dbseclab-v6.2
Jun 10, 2024
681296a
dbseclab-v6.2
Jun 11, 2024
8967071
dbseclab-v6.2
Jun 11, 2024
a4e9945
dbseclab-v6.2
Jun 11, 2024
ab58d4c
dbseclab-v6.2
Jun 11, 2024
04b9513
dbseclab-v6.2
Jun 11, 2024
7848393
dbseclab-v6.2
Jun 11, 2024
c387c96
dbseclab-v6.2
Jun 11, 2024
3af99f5
dv lab updagtes
richardcevans Jun 17, 2024
fd2287a
dv lab updates
richardcevans Jun 17, 2024
6da9a2f
dv lab updates
richardcevans Jun 18, 2024
51827aa
dv lab updates
richardcevans Jun 18, 2024
48763c3
dv lab updates
richardcevans Jun 18, 2024
f92ca47
adb dv lab
richardcevans Jun 18, 2024
1e51f0c
adb dv updates
richardcevans Jun 18, 2024
57d4e96
adb dv lab update
richardcevans Jun 19, 2024
93755f7
dbseclab-v6.2
Jun 19, 2024
92bd32b
Merge branch 'main' of https://github.com/hloumi/security
Jun 19, 2024
57a8420
Merge remote-tracking branch 'upstream/main'
Jun 19, 2024
6b03be9
dbseclab-v6.2
Jun 21, 2024
6e9acdd
dbseclab-v6.2
Jun 21, 2024
c332ddc
dbseclab-v6.2
Jun 21, 2024
ce08910
dbseclab-v6.2
Jun 21, 2024
4a3f7ca
dbseclab-v6.2
Jun 21, 2024
4dd3df0
dbseclab-v6.2
Jun 21, 2024
ace8385
dbseclab-v6.2
Jun 21, 2024
14b2c7d
dbseclab-v6.2
Jun 21, 2024
e6026f2
dv lab updates
richardcevans Jun 21, 2024
102977c
Merge branch 'main' of https://github.com/hloumi/security
richardcevans Jun 21, 2024
d56c1b0
dbseclab-v6.2
Jun 24, 2024
bc94954
Merge branch 'main' of https://github.com/hloumi/security
Jun 24, 2024
82c63cb
dbseclab-v6.2
Jun 24, 2024
7c1d365
Merge branch 'main' into main
anacoman11 Jun 25, 2024
9b368b4
Merge remote-tracking branch 'upstream/main'
Jun 25, 2024
b8d692e
dbseclab-v6.2
Jun 25, 2024
fcbbefa
dbseclab-v6.3
Jun 25, 2024
cd495ff
dbseclab_v6.3
Jun 25, 2024
2120caa
dbseclab_v6.3
Jun 25, 2024
a48b80f
Merge remote-tracking branch 'upstream/main'
Jun 25, 2024
367ec95
dbseclab_v6.3
Jun 25, 2024
4d39592
dbseclab_v6.3
Jul 9, 2024
e42339e
dbseclab_v6.3
Jul 9, 2024
a9d9279
ocw24 dv hol
richardcevans Jul 10, 2024
419858d
dv hol ocw
richardcevans Jul 10, 2024
61d669f
dv ocw hol
richardcevans Jul 10, 2024
96904c0
Merge remote-tracking branch 'upstream/main'
Jul 11, 2024
6f01b9f
dbseclab_v6.3
Jul 12, 2024
6b53d02
Merge remote-tracking branch 'upstream/main'
Jul 12, 2024
28a8eb0
dbseclab_v6.4
Aug 1, 2024
0639719
Merge remote-tracking branch 'upstream/main'
Aug 1, 2024
36e141d
dbseclab_v6.4
Aug 1, 2024
553a644
Merge remote-tracking branch 'upstream/main'
Aug 7, 2024
056b4f3
Merge remote-tracking branch 'upstream/main'
Aug 14, 2024
3a21294
dbseclab_v6.3
Aug 14, 2024
511090c
dbseclab-v6.3
Aug 14, 2024
1a3a66f
dbseclab-v6.3
Aug 14, 2024
f03737a
dbseclab-v6.3
Aug 14, 2024
baf0e08
dbseclab-v6.3
Aug 14, 2024
24cbfaa
dbseclab-v6.3
Aug 14, 2024
7a7ef13
Merge remote-tracking branch 'upstream/main'
Aug 14, 2024
cd54e07
Merge remote-tracking branch 'upstream/main'
Aug 19, 2024
0779993
dbseclab_v6.3.1
Aug 19, 2024
befb63b
Merge remote-tracking branch 'upstream/main'
Aug 20, 2024
db797f1
dbseclab-v6.3.1
Aug 23, 2024
d7cee29
dbseclab-v6.3.1
Aug 23, 2024
fd4a76e
dbseclab-v6.3.1
Aug 23, 2024
2090dc7
dbseclab-v6.3.1
Aug 23, 2024
d86d49d
dbseclab-v6.3.1
Aug 23, 2024
c925085
Merge remote-tracking branch 'upstream/main'
Aug 23, 2024
2e45d42
dbseclab-v6.3.1
Aug 23, 2024
b38fc03
data redaction lab for 23.6 release
richardcevans Sep 23, 2024
2b70fce
free tier folder for new data redaction livelab
richardcevans Sep 23, 2024
42033ee
updated markdown
richardcevans Sep 23, 2024
7c09b23
updates to the data redaction lab
richardcevans Sep 26, 2024
af0e268
Update intro.md
richardcevans Sep 26, 2024
11f77a3
dbsec-v7.0
Sep 26, 2024
c73f406
Merge branch 'main' of https://github.com/hloumi/security
Sep 26, 2024
bd7048c
dbseclab-v7.0
Sep 26, 2024
2337c72
Update manifest.json
richardcevans Sep 26, 2024
aeb2217
Merge remote-tracking branch 'upstream/main'
Nov 4, 2024
acdb672
dbseclab-storyhack_v7.0
Nov 4, 2024
1a233fc
dbseclab-storyhack-v7.0
Nov 4, 2024
575fe01
dbseclab-storyhack_v7.0
Nov 5, 2024
be2d13b
dbseclab_storyhack-v7.0
Nov 5, 2024
8ddd846
dbseclab-storyhack_v7.0
Nov 5, 2024
370c959
dbseclab-storyhack_v7.0
Nov 5, 2024
f42b5be
dbseclab-storyhack_v7.0
Nov 5, 2024
d458f85
Merge remote-tracking branch 'upstream/main'
Dec 16, 2024
a2c88f2
dbseclab_story-v7.1
Dec 16, 2024
31dde63
dbseclab_storyhack-v7.1
Dec 16, 2024
aed8d66
dbseclab_story-hack_v7.1
Dec 16, 2024
637c02b
dbseclab_story-hack_v7.1
Dec 16, 2024
557dbb0
Merge remote-tracking branch 'upstream/main'
Dec 16, 2024
1597b7b
dbseclab_story-hack_v7.1
Dec 16, 2024
aa56553
dbseclab_story-hack_v7.1
Dec 16, 2024
eade8a3
dbseclab_story-hack_v7.1
Dec 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions README_FLND.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# "Freelandia" Demonstration Setup
The following outlines the steps to instantiate a short demonstration showcasing the latest AI-related features of Oracle RDBMS 23ai. The scenario involves the secure querying, including via a conversational interface, of diplomatic communication between the postings of a fictional country ("Freelandia"). The demo uses AI Vector Search and a number of internal PL/SQL packages inside the database to build a RAG pipeline prior to calling out to an external LLM. Label Security is used to enforce access control (row-by-row) on the content that the end-user has access to. This content is provided as context (along with the end-users query) to the external LLM, in order to generate a (hopefully coherent) response.

Standard keyword search (of the same content - but similarly protected by Label Security) is also provided - via Oracle Text. Oracle Database Vault is set up to preclude DBA access to the content, as well as to prevent end-users connecting directly to the database. You will require a 23ai instance with access to the O/S. Note that a new PDB is instantiated as part of the setup.

A viewlet of the demonstration is available [here](https://oradocs.oracle.com/documents/fileview/D7D1774E977684DA23DD8C552F83555D5DF33A10B208/_SecureAIPlatform_Video2_VoiceoverOnly_8m_INTERNAL_ONLY.mkv). A schematic of the solution is shown [here](https://oradocs.oracle.com/documents/fileview/DA556EDBAB0D84D4064F32B6398CD236E7BF7142A822/_SolutionSchematic.pptx).

These setup steps have been tested on the docker/free release of 23ai, but should work in any environment where you have access to the O/S.

Please send any questions to chris.pickett@oracle.com.

## Initial
- Copy and unzip the provided FLND.zip file to some directory ($DIRTOP) on the Oracle 23ai DB server.
- Ensure that the listener.ora file for your 23ai environment has a **static** entry for the CDB. As the setup process involves shutting down/starting up the CDB, this process will fail without a static entry for the CDB in your listener.ora file.
- Edit the parameters.sql file in the $DIRTOP/SQL directory. There is a section at the top where you **must** make changes to certain mandatory parameters, including the location of your Ollama server and the model you wish to use for LLM integration. Parameters in this file marked as optional do not require changing.

## Pluggable Database Creation
- Ensure your environment is setup as follows (ie create and run the following script as the oracle user):

<span style="font-family:Courier">
#!/bin/sh<br>
export ORAENV_ASK=NO<br>
export ORACLE_SID={SID of your CDB}<br>
. /usr/local/bin/oraenv<br>
export ORAENV_ASK=YES<br>
export PATH=$PATH:$ORACLE_HOME/sqlcl/bin
</span>
<br>

- Navigate to $DIRTOP/SQL

- Now execute the following:

<span style="font-family:Courier">
$ sql /nolog<br>
> @create_pdb
</span>
<br>

- Check the create_pdb.lst file - if all has gone well, the end should look like:

<span style="font-family:Courier">
SQL> SELECT * FROM DBA_DV_STATUS;

NAME STATUS
______________________ _________________
DV_CONFIGURE_STATUS TRUE
DV_ENABLE_STATUS TRUE
DV_APP_PROTECTION NOT CONFIGURED

SQL> SELECT * FROM DBA_OLS_STATUS;

NAME STATUS DESCRIPTION
_______________________ _________ __________________________________
OLS_CONFIGURE_STATUS TRUE Determines if OLS is configured
OLS_ENABLE_STATUS TRUE Determines if OLS is enabled
</span>

## Schema Creation
- From the same directory as above (i.e. $DIRTOP/SQL), execute the following:

<span style="font-family:Courier">
$ sql /nolog<br>
> @create_all
</span>
<br>

- Check the create_all.lst file for any errors. Some errors may be due to attempts to drop non-existent objects. These can be ignored.
- Use the testing.sql file in the $DIRTOP/SQL directory to test asking questions and receiving answers from the LLM.

## Python (Streamlit) Setup
- Coming

## TODO
- Text/AI: Load all the attachments (specific to each cable) into attachment table
- Unified Index: Unify Oracle Text and vector indices (requires 23.6)
- DBV: DB Vault connect rules to create MAC-realm around tables, and CONNECT rules (disallow) for end-users not connecting through Python app
- AVDF: Unified audit rules for SELECTs on chunk table and invocations of response function - capture user's NL query and (potentially) use vector search (somewhere?) to detect early IoC based on a user's "strange" question...
- Spatial: Geo-coding of missions (postings), to allow spatially-based analysis of trends
- JSON: Actual use of JSON/Relational duality (not currently possible due to validation-related error on ingestion) rather than current approach of loading JSON document into a JSON column followed by manual shredding into separate tables. This is why I have kept the approach of loading (synthetically-generated) JSON documents (in the $DIRTOP/JSON directory) when loading the data, as the intent is to move eventually to a duality-based approach.
- Access Ctrl: Higher-level analysis/summary available to analysts *without* them having access to lower-level data (common use case in Justice & Public Safety domains)
- Low-code UIs: APEX and VisualBuilder/JET, in addition to Python
- Autonomous: Equivalent setup scripts for ADB
Loading