fix: fix for ingest non-English file names and URLs

[Akhileswara-Microsoft]

Purpose

This pull request introduces improvements to how filenames and URLs are handled throughout the backend, focusing on consistent encoding and decoding of non-ASCII characters (such as those in Hebrew) to ensure compatibility and better user experience. The main changes involve encoding filenames and URLs before storage or upload, and decoding them for display in the UI, as well as adding user-agent headers to HTTP requests for better compatibility with external sites.

Filenames and URLs encoding/decoding improvements:

• Filenames and URLs are now encoded using urllib.parse.quote before being uploaded to blob storage or used as metadata, ensuring non-ASCII characters are handled correctly and URL structure is preserved (add_url_embeddings.py, 01_Ingest_Data.py). [1] [2]
• When displaying filenames in the UI (file selection in Explore and Delete pages), filenames are decoded using urllib.parse.unquote so users see readable names instead of encoded strings (02_Explore_Data.py, 03_Delete_Data.py). [1] [2]
• Deleted files are now shown with decoded names in the success message after deletion, improving clarity for users (03_Delete_Data.py).

HTTP request improvements:

• Added a User-Agent header to outgoing requests in download_url_and_upload_to_blob to avoid being blocked by certain websites (e.g., Wikipedia), and ensured URLs are encoded before making the request.

Imports and code hygiene:

• Added necessary imports of urllib.parse in affected files to support encoding/decoding operations. [1] [2] [3]

Does this introduce a breaking change?

• Yes
• No

How to Test

• Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

What to Check

Verify that the following are valid
upload file or url with name in non-english

• ...

[Copilot]

Pull request overview

This pull request introduces encoding and decoding of non-ASCII characters in filenames and URLs to support non-English content (e.g., Hebrew characters) throughout the application. The changes aim to ensure that filenames are properly encoded when stored in blob storage and metadata, and decoded when displayed in the UI.

Changes:

• Added URL encoding using urllib.parse.quote for filenames and URLs before storage in blob storage and metadata
• Added URL decoding using urllib.parse.unquote for filenames when displaying them in UI (Explore Data and Delete Data pages)
• Added User-Agent header to HTTP requests in URL ingestion to avoid being blocked by websites

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 11 comments.

File	Description
code/backend/batch/add_url_embeddings.py	Added URL encoding for blob names and metadata, plus User-Agent header for HTTP requests
code/backend/pages/01_Ingest_Data.py	Added URL encoding for metadata title, commented out previous sanitization function
code/backend/pages/02_Explore_Data.py	Added filename decoding for display in file selection dropdown with encoded-to-decoded mapping
code/backend/pages/03_Delete_Data.py	Added filename decoding for checkboxes and success messages after deletion

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.