Release 0.6.0 (#286)

* Refactor code to remove isExecuted property in NotebookModel.js and Cell.js, and update dependencies in package.json (#228)

* Update notebook execution timestamps and clear cell outputs (#229)

* Update notebook execution timestamps and fix styling in TextResult.js (#230)

* Update notebook execution timestamps and fix styling in TextResult.js, refactor code to remove isExecuted property in NotebookModel.js and Cell.js, update dependencies in package.json, merge 0.5.2, refactor notebook execution code and add keyboard shortcut for running cells, refactor styling and add MoveButton component in NotebookHeader.js, refactor naming conventions in Notebook.js, NotebookTitle.js, DeleteDialog.js, Item.js, and RenameDialog.js, refactor styling in CreateFolderDialog.js, CreateNotebookDialog.js, WorkspaceSidebar.js, and Item.js. (#232)

* 233 create notebook from server (#236)

* Update notebook execution timestamps and fix styling in TextResult.js, refactor code to remove isExecuted property in NotebookModel.js and Cell.js, update dependencies in package.json, refactor notebook execution code and add keyboard shortcut for running cells, refactor styling and add MoveButton component in NotebookHeader.js, refactor naming conventions in Notebook.js, NotebookTitle.js, DeleteDialog.js, Item.js, and RenameDialog.js, refactor styling in CreateFolderDialog.js, CreateNotebookDialog.js, WorkspaceSidebar.js, and Item.js.

* Update notebook execution timestamps and fix styling in TextResult.js, refactor code to remove isExecuted property in NotebookModel.js and Cell.js, update dependencies in package.json, refactor notebook execution code and add keyboard shortcut for running cells, refactor styling and add MoveButton component in NotebookHeader.js, refactor naming conventions in Notebook.js, NotebookTitle.js, DeleteDialog.js, Item.js, and RenameDialog.js, refactor styling in CreateFolderDialog.js, CreateNotebookDialog.js, WorkspaceSidebar.js, and Item.js.

* Update docker-compose.yaml to mount init.sql file for PostgreSQL container (#238)

* Update notebook routes and services to include delete functionality (#240)

* Update notebook routes and services to include get_notebook_by_path functionality (#243)

* Update notebook routes and services to include logging and notebook renaming functionality (#245)

* 244 rename directory from server (#247)

* Add directories table to PostgreSQL init.sql script

* Update notebook routes and services to include directory functionality

* 249 add integration test (#250)

* Add Docker build workflow for server and update .gitignore

* Add requirements.txt to Dockerfile

* Update Dockerfile to remove copying of requirements.txt

* Add server service to docker-compose.yaml and update server Dockerfile

* Update server run.py to bind to all network interfaces

* Update server run.py to use 'postgres' as the hostname for the PostgreSQL database

* Update test.

* Add environment variable for testing in docker-compose.yaml

* Update Jupyter server path in server config

* Fix typo in Notebook class

* Update Jupyter server path in server config and fix typo in Notebook class

* Update Jupyter server path in server config and fix typo in Notebook class

* Update notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in NotebookModel.js, App.js, notebook.py, and directory.py

* Update notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in NotebookModel.js, App.js, and notebook.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in NotebookModel.js, App.js, notebook.py, and directory.py

* Add integration test workflow

* Add integration test workflow and update badges in README.md

* Add integration test workflow and update badges in README.md

* Update ENV variable in integration test workflow

* Add unit tests for DirectoryModel in test_directory_model.py and remove unused import in test_notebook_model.py

* Update ENV variable in integration test workflow and server config

* Add unit tests for NotebookService in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName'

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Add print statement to test_notebook_service.py

* Add notebook_1 creation to test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model in test_notebook_service.py

* Refactor notebook creation API endpoint and model to log response in test_notebook_service.py

* Refactor notebook creation API endpoint and model to handle error response from Jupyter Server

* Refactor notebook creation API endpoint and model to use relative paths in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use relative paths and update test_notebook_service.py

* Refactor notebook creation API endpoint and model to use empty Jupyter token and password

* Update Content-Security-Policy and X-Frame-Options in jupyter_notebook_config.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor Docker options and command in integration-test.yml

* Update Jupyter notebook Dockerfile to include command for running Jupyter notebook with specific options

* Refactor Docker options and command in integration-test.yml

* Update Jupyter server port in integration-test.yml and config.py

* Update Jupyter server port in integration-test.yml to use port 6666 instead of 8889

* Update Jupyter server port in integration-test.yml and config.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py

* Refactor print statement in test_notebook_service.py to use relative path instead of absolute path

* Refactor print statement in test_notebook_service.py to use relative path instead of absolute path

* Refactor print statement in test_notebook_service.py to use relative path instead of absolute path

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model to use 'name' and 'path' instead of 'notebookName' in test_notebook_service.py and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model in test_notebook_service.py, and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model in test_notebook_service.py, and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model in test_notebook_service.py, and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model in test_notebook_service.py, and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model in test_notebook_service.py, and fix test_create_and_get_notebook test case

* Refactor notebook creation API endpoint and model in test_notebook_service.py, fix test_create_and_get_notebook test case, and update return statement in Notebook class

* Refactor notebook creation API endpoint and model in test_notebook_service.py, fix test_create_and_get_notebook test case, and update return statement in Notebook class

* Refactor notebook creation API endpoint and model in test_notebook_service.py, fix test_create_and_get_notebook test case, and handle non-existent notebook in Notebook class

* Refactor notebook creation API endpoint and model in test_notebook_service.py, fix test_create_and_get_notebook test case, and update print statement in Notebook class

* Refactor notebook creation API endpoint and model in test_notebook_service.py, fix test_create_and_get_notebook test case, and handle non-existent notebook in Notebook class

* Refactor notebook creation API endpoint and model in test_notebook_service.py, fix test_create_and_get_notebook test case, and update notebook retrieval in Notebook class

* Refactor directory service to include get_content_by_path method and add unit test for it

* Refactor directory service to include default path handling in get_content_by_path method

* Refactor directory service to include get_content_by_path method and update unit test

* Refactor directory service to handle None directory path in get_content_by_path method

* Refactor directory service to include test_create_directory method and update unit test

* Refactor test_get_content_by_path to test_create_directory in test_directory_service.py

* Refactor test_create_directory method and update unit test in test_directory_service.py

* Refactor test_create_directory method and update unit test in test_directory_service.py

* Refactor test_create_directory method and update unit test in test_directory_service.py

* Refactor test_rename_directory_by_path method and update unit test in test_directory_service.py

* Fix error handling when renaming directory in DB

* Refactor test_rename_directory_by_path method and update unit test in test_directory_service.py

* Refactor test_rename_directory_by_path method and update unit test in test_directory_service.py

* Update get_content_by_path method in test_directory_service.py to fix JSON parsing issue

* Fix JSON parsing issue in get_content_by_path method

* Fix JSON parsing issue in get_content_by_path method in test_directory_service.py

* Fix JSON parsing issue in get_content_by_path method in test_directory_service.py

* Refactor test_directory_service.py to fix JSON parsing issue and improve directory renaming logic

* Fix JSON parsing issue and add print statement in test_directory_service.py

* Fix JSON parsing issue and update print statement in test_directory_service.py

* Refactor test_directory_service.py to fix JSON parsing issue and update print statement

* Fix JSON parsing issue and update print statement in test_directory_service.py

* Fix JSON parsing issue and update print statement in test_directory_service.py

* Fix JSON parsing issue and update Flask response in directory.py

* Fix JSON parsing issue and update Flask response in directory.py

* Refactor test_directory_service.py to fix JSON parsing issue and update Flask response

* Fix JSON parsing issue and update print statement in test_directory_service.py

* Update print statement in test_directory_service.py

* Fix JSON parsing issue and update Flask response in directory.py

* Update print statement in test_directory_service.py

* Update print statement in test_directory_service.py

* Fix JSON parsing issue and update Flask response in directory.py

* Update JSON parsing and response handling in test_directory_service.py

* Update JSON parsing and response handling in test_directory_service.py

* Refactor JSON parsing and response handling in test_directory_service.py

* Update status code assertion in test_directory_service.py

* Refactor JSON parsing and response handling in test_directory_service.py

* Update print statement in test_directory_service.py

* Update print statement and add debug output in test_directory_service.py

* Refactor print statement and add debug output in test_directory_service.py

* Update directory path assignment in directory.py

* Refactor directory path assignment and update content retrieval in test_directory_service.py

* Refactor directory path assignment and update content retrieval in test_directory_service.py

* Update logger statement in directory.py

* Refactor directory creation and retrieval in directory.py and test_directory_service.py

* Fix directory renaming bug in directory.py

* Refactor directory path assignment and update content retrieval in test_directory_service.py

* Add print statement in test_notebook_service.py

* Update print statement in test_notebook_service.py

* Refactor notebook retrieval in notebook.py

* Fix response handling in Notebook class

* Refactor response handling in Notebook class

* Update print statement in test_notebook_service.py

* Refactor response handling in Notebook class and fix response decoding bug in notebook.py

* Refactor response handling in Notebook class and fix response decoding bug in notebook.py

* Refactor response handling in Notebook class and fix response decoding bug in notebook.py

* Refactor response handling in Notebook class and fix response decoding bug in notebook.py

* Refactor response handling in Notebook class and fix response decoding bug in notebook.py

* Refactor response handling and fix response decoding bug in notebook.py

* Fix error handling in Notebook class when getting notebook from Jupyter Server

* Update print statement in test_notebook_service.py

* Refactor response handling and fix response decoding bug in Notebook class

* Update print statement in test_notebook_service.py

* Refactor response handling and fix response decoding bug in Notebook class

* Add test case for creating and getting notebook by path in test_notebook_service.py

* Add test case for deleting notebooks in test_notebook_service.py

* Refactor delete_notebook test case in test_notebook_service.py

* Refactor notebook path handling in Notebook class

* Add test case for renaming notebooks in test_notebook_service.py

* Rename notebook in Notebook class

* Update Flask response in directory.py to return JSON response with correct mimetype

* 252 fix create notebook (#253)

* Update notebook routes and services to include new notebook creation and retrieval functionality

* Fix status code assertion in test_notebook_service.py

* Update notebook creation logic to ensure notebook names end with '.ipynb'

* Update status code assertion in test_notebook_service.py

* 234 update notebook from server (#254)

* Update Notebook class to include update_notebook method in notebook.py

* Update Notebook class to include update_notebook method in notebook.py

* Migrate save notebook to my own BE.

* Update NotebookModel.js to remove commented out code block

* Update import path in RunButton.test.js

* Update Notebook class to include nbformat and nbformat_minor in notebook.py

* Update Notebook class to include nbformat and nbformat_minor in notebook.py

* Update notebook execution status and add new code cell in notebook_20240709183641.ipynb

* Update notebook_20240709183641.ipynb, notebook.py, and test_notebook_service.py

* 242 move notebook from server (#260)

* Add test for move notebook.

* Update test_notebook_service.py to include test for moving a notebook to a non-existent path

* Update NotebookModel.js to include moveNotebook method

* Fix notebook rename. (#262)

* Update notebook and rename dialog components (#263)

* Update directory routes and services to include get_directory_content functionality (#266)

* 264 call be to delete directory (#267)

* Refactor directory routes and services to include get_directory_content functionality and update Flask response in directory.py to return JSON response with correct mimetype

* Update status code assertion in test_directory_service.py

* Refactor directory routes and services to include delete_directory functionality

* 258 call be to create session (#270)

* Update session service to include create_session method in session.py

* Update print statement in test_session_service.py

* Refactor test_session_service.py and session.py

* Update Jupyter API paths in directory.py, notebook.py, and session.py

* Refactor test_session_service.py to include session creation and kernel validation

* Update Jupyter API paths and add session creation functionality

* 268 call be to get session (#271)

* The developer is trying to commit the following changes to the repository:

1. In the `server/app/services/session.py` file, a new method `get_all_sessions()` is added to the `Session` class. This method makes a GET request to the Jupyter Session API and returns the response as a JSON object.

2. In the `server/tests/services/test_session_service.py` file, a new test case `test_get_all_sessions()` is added to the `SessionServiceTestCase` class. This test case calls the `get_all_sessions()` method and asserts that the response status code is 200.

3. In the `webapp/src/components/notebook/Notebook.js` file, the `Notebook` component is updated. The `getSession()` method from the `NotebookModel` class is replaced with the `getSession()` method from the `SessionModel` class.

4. In the `webapp/src/models/NotebookModel.js` file, the `getSession()` method is removed and replaced with the `getSession()` method from the `SessionModel` class.

5. In the `webapp/src/models/SessionModel.js` file, the `getSession()` method is added to the `SessionModel` class. This method makes a GET request to the Jupyter Session API and returns the kernel ID associated with the given notebook path.

These changes are related to adding functionality to retrieve all sessions from the Jupyter Server and updating the `Notebook` component to use the `SessionModel` class for session-related operations.

* Update session service to include create_session method in session.py, refactor test_session_service.py and session.py, and add session creation functionality

* Update session service to include get_session_by_path method in session.py and add test_get_session_by_path method in test_session_service.py

* Update session service to include response as JSON in session.py

* Update session service to include get_all_sessions method in session.py and update Notebook.js and SessionModel.js to use the new method

* 256 call be to restart kernel (#273)

* Update kernel service to include restart_kernel method in kernel.py

* Update kernel service to include setUp method in test_kernel_service.py

* Update assertion in test_kernel_service.py to use utf-8 decoding for response data

* Update response content in kernel.py and session.py to use response.content instead of response in the return statement

* Update response content in kernel.py and session.py, and assertion in test_kernel_service.py

* Update response content in kernel.py and session.py, and assertion in test_kernel_service.py

* Update response content in kernel.py and session.py, and assertion in test_kernel_service.py

* Refactor test_kernel_service.py to use response.content instead of response in the return statement

* Refactor test_kernel_service.py to use response.content instead of response in the return statement

* Refactor test_kernel_service.py to use response.content instead of response in the return statement

* Update assertion in test_kernel_service.py to use response.content instead of response in the return statement

* Add kernel restart endpoint to kernel.py and update references in related files

* 272 call be to get kernel (#274)

* Update Kernel class to include get_kernel_by_id method in kernel.py, add test_get_kernel_by_id method in test_kernel_service.py, and update NotebookKernel.js to import KernelModel from the correct path

* Update Kernel class to include get_kernel_by_id method, update NotebookKernel.js to import KernelModel from the correct path, and refactor KernelModel to remove basePath parameter in restartKernel method

* Update environment variables and API endpoints in webapp and server code (#275)

* 276 store spark info into db (#277)

* Update notebook and server code to include SparkAppModel and SparkApp service (#276)

* Update SparkApp service to include create_spark_app method in spark_app.py and add test_create_spark_app method in test_spark_app_service.py

* Update SparkAppModel primary key in spark_app.py

* Update SparkApp service to include create_spark_app method in spark_app.py and add test_create_spark_app method in test_spark_app_service.py

* Update SparkApp service to include create_spark_app method and test case in test_spark_app_service.py

* Update SparkApp service to fix encoding issue in test_spark_app_service.py

* Add print statement to test_spark_app_service.py for debugging purposes

* Fix notebook_path in test_create_spark_app method of test_spark_app_service.py

* Remove print statement in test_spark_app_service.py

* Add NotebookModel import and update notebook spark_app_id check in test_spark_app_service.py

* Update SparkApp service to include get_spark_app_by_notebook_path method in notebook.py and add test_get_spark_app_by_notebook_path method in test_notebook_service.py

* Add database tables for spark_apps, directories, and notebooks in init.sql

* Update NotebookModel and NotebookSparkAppModel to remove spark_app_id column and update related code

* Fix spark_app_dict assertion in test_spark_app_service.py

* Update notebook_id assignment in SparkApp service to use the 'id' attribute instead of 'notebook_id'

* Refactor SparkApp service to create and update spark app in database

* Refactor SparkApp service to associate spark app with notebook in database

* Refactor SparkApp service to use 'id' attribute instead of 'notebook_id' for notebook_id assignment

* Refactor SparkApp service to use NotebookModel query instead of Notebook.get_notebook_by_path in test_spark_app_service.py

* Refactor NotebookSparkAppModelTestCase to use NotebookModel and SparkAppModel in test_notebook_spark_app_model.py

* Fix error handling in Notebook.get_spark_app_by_notebook_path method

* Refactor Notebook.get_spark_app_by_notebook_path method to use 'id' attribute instead of 'notebook_id' for notebook_id assignment

* Refactor Notebook.get_spark_app_by_notebook_path method to use 'id' attribute instead of 'notebook_id' for notebook_id assignment

* Refactor Notebook.get_spark_app_by_notebook_path method to use 'id' attribute instead of 'notebook_id' for notebook_id assignment

* Refactor Notebook.get_spark_app_by_notebook_path method to use 'id' attribute for notebook_id assignment and update test cases

* Refactor database initialization script and update notebook names in init.sql

* Update SparkAppModel and SparkApp service to store spark app information in the database (#279)

* Update init.sql to include new notebooks and directories, and update Runs.js to display a message when there are no Spark Applications associated with the notebook (#281)

* Update environment variables and API endpoints in webapp and server code, and add Spark UI base URL to config.js (#283)

* 284 integrate airflow to webapp (#285)

* Update environment variables and API endpoints in webapp and server code, and add Spark UI and Airflow base URLs to config.js and .env.dev

* Update nginx.conf to proxy_pass to airflow-webserver instead of history-server and update X-Frame-Options header in docker/nginx/nginx.conf

* Update nginx.conf, environment variables, and API endpoints, and add Spark UI and Airflow base URLs

* Update environment variables and API endpoints, and add Spark UI and Airflow base URLs to webapp and server code

Author: xuwenyihust

.github/workflows/build-docker-server.yml

@@ -0,0 +1,28 @@
+name: Build Docker - Server
+
+on:
+  push:
+    paths:
+      - 'server/**'
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@v1
+        with:
+            username: ${{ secrets.DOCKERHUB_USERNAME }}
+            password: ${{ secrets.DOCKERHUB_PASSWORD }}
+            
+      - name: Docker Build & Push
+        run: |
+          timestamp=$(date +"%Y%m%d%H%M%S")
+          
+          docker build -t ${{ secrets.DOCKERHUB_USERNAME }}/server:$timestamp -f server/Dockerfile ./server
+          docker tag ${{ secrets.DOCKERHUB_USERNAME }}/server:$timestamp ${{ secrets.DOCKERHUB_USERNAME }}/server:latest
+
+          docker push ${{ secrets.DOCKERHUB_USERNAME }}/server:$timestamp
+          docker push ${{ secrets.DOCKERHUB_USERNAME }}/server:latest
+

.github/workflows/integration-test.yml

@@ -0,0 +1,49 @@
+name: Integration Test
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    services:
+      postgres:
+        image: postgres:13
+        env:
+          POSTGRES_USER: server
+          POSTGRES_PASSWORD: password-server
+          POSTGRES_DB: server_db
+        ports:
+          - 5432:5432
+        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
+
+      notebook:
+        image: wenyixu101/notebook:latest
+        env:
+          JUPYTER_ENABLE_LAB: "no"
+        ports:
+          - "8888:8888"
+        options: --health-cmd "curl -f http://localhost:8888 || exit 1" --health-interval 10s --health-timeout 5s --health-retries 5
+        
+        
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.11'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+        working-directory: server
+
+      - name: Run tests
+        run: |
+          python -m unittest discover tests
+        working-directory: server
+        env:
+          ENV: integration

.gitignore

@@ -208,3 +208,4 @@ webapp/yarn-debug.log*
 webapp/yarn-error.log*
 webapp/node_modules/*
 node_modules/*
+.DS_Store

README.md

@@ -10,6 +10,7 @@
 ![GitHub Repo stars](https://img.shields.io/github/stars/xuwenyihust/datapulse)
 ![Apache Spark 3.5.0](https://img.shields.io/badge/Apache%20Spark-3.5.0-brightgreen?logo=apachespark)
 [![GitHub License](https://img.shields.io/github/license/xuwenyihust/Data-Platform?label=License)](https://github.com/xuwenyihust/Data-Platform/blob/main/LICENSE)
+[![Integration Test](https://github.com/xuwenyihust/PawMark/actions/workflows/integration-test.yml/badge.svg)](https://github.com/xuwenyihust/PawMark/actions/workflows/integration-test.yml)
 [![Test Web App](https://github.com/xuwenyihust/PawMark/actions/workflows/test-webapp.yml/badge.svg)](https://github.com/xuwenyihust/PawMark/actions/workflows/test-webapp.yml)
 
 ## Summary
@@ -77,6 +78,15 @@ PawMark is a platform for big data and AI. It is based on Apache Spark and Kuber
 
 </details>
 
+<details>
+<summary>Server</summary>
+
+[![Build Docker - Server](https://github.com/xuwenyihust/PawMark/actions/workflows/build-docker-server.yml/badge.svg)](https://github.com/xuwenyihust/PawMark/actions/workflows/build-docker-server.yml)
+
+- [Dockerfile](./server/Dockerfile) 
+
+</details>
+
 <details>
 <summary>Spark</summary>
 
@@ -131,6 +141,7 @@ PawMark is a platform for big data and AI. It is based on Apache Spark and Kuber
 ## Versions
 <details>
 <summary>Details</summary>
+
 | Component    | Version |
 |--------------|---------|
 | Scala        | 2.12    |
@@ -142,6 +153,7 @@ PawMark is a platform for big data and AI. It is based on Apache Spark and Kuber
 | Airflow      | 2.9.1   |
 | Postgres     | 13      |
 | React        | 18.3.1  |
+
 </details>
 
 ## License

docker-compose.yaml

@@ -13,6 +13,20 @@ services:
           cpus: "1" 
           memory: 1g
 
+  server:
+    image: wenyixu101/server:latest
+    container_name: server
+    ports:
+      - "5002:5002"
+    environment:
+      - ENV=testing
+    pull_policy: always
+    deploy:
+      resources:
+        limits:
+          cpus: "1" 
+          memory: 1g
+
   notebook:
     image: wenyixu101/notebook:latest
     container_name: notebook
@@ -23,7 +37,7 @@ services:
       - JUPYTER_ENABLE_LAB=no
       - ENVIRONMENT=development
       - PYSPARK_SUBMIT_ARGS=--packages io.delta:delta-spark_2.12:3.0.0 --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog pyspark-shell
-    command: ["jupyter", "notebook", "--ip='0.0.0.0'", "--port=8888", "--no-browser", "--allow-root", "--NotebookApp.token=''", "--NotebookApp.password=''"]
+    # command: ["jupyter", "notebook", "--ip='0.0.0.0'", "--port=8888", "--no-browser", "--allow-root", "--NotebookApp.token=''", "--NotebookApp.password=''"]
     volumes:
       - ./data/spark-events:/opt/data/spark-events
       - ./data/spark-warehouse:/opt/data/spark-warehouse
@@ -185,7 +199,8 @@ services:
     ports:
       - "5432:5432"
     volumes:
-      - ./data/postgres:/var/lib/postgresql/data
+      # - ./data/postgres:/var/lib/postgresql/data
+      - ./docker/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql
     pull_policy: always
     deploy:
       resources:

docker/airflow/Dockerfile

@@ -18,3 +18,6 @@ RUN python3 -m ipykernel install --user --name python3 --display-name "Python 3"
 # Set JAVA_HOME environment variable
 ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
 
+COPY webserver_config.py /opt/airflow/webserver_config.py
+COPY airflow.cfg /opt/airflow/airflow.cfg
+